Real-time detection of lanes and boundaries by autonomous vehicles

ABSTRACT

In various examples, sensor data representative of an image of a field of view of a vehicle sensor may be received and the sensor data may be applied to a machine learning model. The machine learning model may compute a segmentation mask representative of portions of the image corresponding to lane markings of the driving surface of the vehicle. Analysis of the segmentation mask may be performed to determine lane marking types, and lane boundaries may be generated by performing curve fitting on the lane markings corresponding to each of the lane marking types. The data representative of the lane boundaries may then be sent to a component of the vehicle for use in navigating the vehicle through the driving surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 17/222,680, filed Apr. 5, 2021, which is acontinuation of U.S. patent application Ser. No. 16/286,329, filed Feb.26, 2019, which claims the benefit of U.S. Provisional Application No.62/636,142, filed on Feb. 27, 2018. Each of these applications isincorporated herein by reference in its entirety.

BACKGROUND

For autonomous vehicles to operate safely in all environments, theautonomous vehicles must be capable of effectively performing vehiclemaneuvers—such as lane keeping, lane changing, lane splits, turns,stopping and starting at intersections, crosswalks, and the like, and/orother vehicle maneuvers. For example, for an autonomous vehicle tonavigate through surface streets (e.g., city streets, side streets,neighborhood streets, etc.) and on highways (e.g., multi-lane roads),the autonomous vehicle is required to navigate an often rapidly movingvehicle among one or more divisions (e.g., lanes, intersections,crosswalks, boundaries, etc.) of a road that are often minimallydelineated, and may be difficult to identify in certain conditions evenfor the most attentive and experienced of drivers. In other words, anautonomous vehicle is required to be a functional equivalent of anattentive human driver, who draws upon a perception and action systemthat has an incredible ability to identify and react to moving andstatic obstacles in a complex environment, merely to avoid collidingwith other objects or structures along its path.

Conventional approaches to detecting lane and road boundaries includegenerating and processing images from one or more cameras, andattempting to interpolate the lane and road boundaries from visualindicators identified during the processing (e.g., using computingvision or other machine learning techniques. However, performing laneand road boundary detection in this way has proven to be either toocomputationally expensive to run effectively in real-time and/or hassuffered from inaccuracy as a result of shortcuts implemented to reducecomputing requirements. In other words, these conventional systemseither forego accuracy to operate in real-time, or forego operation inreal-time to produce acceptable accuracy. Additionally, even inconventional systems that achieve a level of accuracy required for safeand effective operation of autonomous vehicles, the accuracy is limitedto ideal road and weather conditions. As a result, autonomous vehiclesthat operate using these conventional approaches may not be able toaccurately operate in real-time and/or with accuracy in all road andweather conditions.

SUMMARY

Embodiments of the present disclosure relate to using machine learningmodels to detect lanes and road boundaries by autonomous vehicles andadvanced driver assistance systems in real-time. More specifically,systems and methods are disclosed that provide for accurate detectionand identification of lanes and road boundaries in real-time using adeep neural network that is trained—e.g., using low-resolution images,region of interest images, and a variety of ground truth masks—to detectlanes and boundaries in a variety of situations, including less thanideal weather and road conditions.

In contrast to conventional systems, such as those described above, thecurrent system may use one or more machine learning models that arecomputationally inexpensive and capable of real-time deployment todetect lanes and boundaries. The machine learning model(s) may betrained with a variety of annotations as well as a variety oftransformed images such that the machine learning model(s) is capable ofdetecting lanes and boundaries in an accurate and timely manner,especially at greater distances. The machine learning model(s) may betrained using low-resolution images, region of interest images (e.g.,cropped images), transformed images (e.g., spatially augmented, coloraugmented, etc.), ground truth labels or masks, and/or transformedground truth labels or masks (e.g., augmented according to thecorresponding augmentation of the transformed images to which theyrelate). The machine learning model(s) may also be trained using bothbinary and multi-class segmentation masks, further increasing theaccuracy of the model. In addition, post-processing may be performed onoutputs of the machine learning model(s) to more accurately identify andlabel types and contours of the lane markings and boundaries. Afterpost-processing, lane curves and labels may be generated that may beused by one or more layers of an autonomous driving software stack—suchas a perception layer, a world model management layer, a planning layer,a control layer, and/or an obstacle avoidance layer.

As a result of executing lane and road boundary detection according tothe processes of the present disclosure, autonomous vehicles may be ableto detect lanes and road boundaries of a driving surface to effectivelyand safely navigate within a current lane, through lane changes, throughlane merges and lane splits, through intersections, and/or through otherfeatures of the driving surface in a variety of road and weatherconditions. In addition, because of the architecture of the machinelearning model(s), the training methods for the machine learningmodel(s), and the post-processing methods for converting outputs of themachine learning model(s) to lane curves and labels, lane and boundarydetection performed according to the present disclosure may be lesscomputationally expensive—requiring less processing power, energyconsumption, and bandwidth—than in conventional approaches.

BRIEF DESCRIPTION OF THE DRAWINGS

The present systems and methods for real-time detection of lanes androad boundaries by autonomous vehicles is described in detail below withreference to the attached drawing figures, wherein:

FIG. 1A is a data flow diagram illustrating an example process fordetecting lanes and road boundaries, in accordance with some embodimentsof the present disclosure;

FIG. 1B is an illustration of an example machine learning model, inaccordance with some embodiments of the present disclosure;

FIG. 1C is an illustration of another example machine learning model, inaccordance with some embodiments of the present disclosure;

FIG. 2 is a flow diagram illustrating a method for detecting lanes androad boundaries, in accordance with some embodiments of the presentdisclosure;

FIG. 3A is a data flow diagram illustrating an example process fortraining a machine learning model(s) to detect lanes and roadboundaries, in accordance with some embodiments of the presentdisclosure;

FIG. 3B is a data flow diagram illustrating an example process forgenerating training images to train a machine learning model(s), inaccordance with some embodiments of the present disclosure;

FIG. 3C includes a data flow diagram illustrating an example process fortraining a machine learning model(s) using a multi-class mask headand/or a binary mask head, in accordance with some embodiments of thepresent disclosure;

FIG. 3D is a flow diagram illustrating a method for training a machinelearning model(s) to detect lanes and road boundaries using transformedimages and transformed labels as ground truth data, in accordance withsome embodiments of the present disclosure;

FIG. 4A is a data flow diagram illustrating an example process forgenerating ground truth data to train a machine learning model(s) todetect lanes and road boundaries, in accordance with some embodiments ofthe present disclosure;

FIG. 4B is a data flow diagram illustrating an example process forperforming data augmentation and cropping of ground truth masks, inaccordance with some embodiments of the present disclosure;

FIG. 4C is a flow diagram illustrating a method for training a machinelearning model(s) to detect lanes and road boundaries using down-sampledimages and/or ground truth masks, in accordance with some embodiments ofthe present disclosure;

FIG. 4D is a flow diagram illustrating a method for training a machinelearning model(s) to detect lanes and road boundaries using croppedimages and/or ground truth masks, in accordance with some embodiments ofthe present disclosure;

FIG. 5A is an illustration of an example process for annotating roadboundaries for ground truth data, in accordance with some embodiments ofthe present disclosure;

FIG. 5B is an illustration of an example road boundary annotation, inaccordance with some embodiments of the present disclosure;

FIG. 5C is a flow diagram illustrating a method for annotating roadboundaries for ground truth generation, in accordance with someembodiments of the present disclosure;

FIG. 6A is an illustration of an example crosswalk and intersectionannotation, in accordance with some embodiments of the presentdisclosure;

FIGS. 6B and 6C are diagrams illustrating example lane mergeannotations, in accordance with some embodiments of the presentdisclosure;

FIGS. 6D and 6E are diagrams illustrating example lane splitannotations, in accordance with some embodiments of the presentdisclosure;

FIG. 7A is an illustration of example performance calculations atdifferent regions of a training image, in accordance with someembodiments of the present disclosure;

FIG. 7B is an example illustration of a two-dimensional (2D) KPImeasurement using lane detection and ground truth polyline points, inaccordance with some embodiments of the present disclosure;

FIG. 7C is a diagram illustrating a three-dimensional (3D) KPImeasurement using lane detection and ground truth polyline points, inaccordance with some embodiments of the present disclosure;

FIG. 8A is an illustration of an example autonomous vehicle, inaccordance with some embodiments of the present disclosure;

FIG. 8B is an example of camera locations and fields of view for theexample autonomous vehicle of FIG. 8A, in accordance with someembodiments of the present disclosure;

FIG. 8C is a block diagram of an example system architecture for theexample autonomous vehicle of FIG. 8A, in accordance with someembodiments of the present disclosure;

FIG. 8D is a system diagram for communication between cloud-basedserver(s) and the example autonomous vehicle of FIG. 8A, in accordancewith some embodiments of the present disclosure; and

FIG. 9 is an example block diagram for an example computing devicesuitable for implementation of embodiments of the present disclosure.

DETAILED DESCRIPTION

Systems and methods are disclosed related to using one or more machinelearning models to detect, in real-time, lanes and road boundaries byautonomous vehicles and/or advanced driver assistance systems (ADAS).The present disclosure may be described with respect to an exampleautonomous vehicle 800 (alternatively referred to herein as “vehicle800” or “autonomous vehicle 800”, an example of which is describedherein with respect to FIGS. 8A-8D. However, this is not intended to belimiting. For example, the systems and methods described herein may beused in augmented reality, virtual reality, robotics, and/or othertechnology areas, such as for localization, calibration, and/or otherprocesses. In addition, although the detections described herein relateprimarily to lanes, road boundaries, lane splits, lane merges,intersections, crosswalks, and/or the like, the present disclosure isnot intended to be limited to only these detections. For examples, theprocesses described herein may be used for detecting other objects orfeatures, such as signs, poles, trees, barriers, and/or other objects orfeatures. In addition, although the description in the presentdisclosure separates lane detections from lane splits and lane merges,this is not intended to be limiting. For example, features andfunctionality described herein with respect to detecting lanes and roadboundaries may also be applicable to detecting lane splits and/or lanemerges. In the alternative, features and functionality described hereinwith respect to detecting lane splits and/or lane merges may also beapplicable to detecting lanes and/or road boundaries.

Lane and Road Boundary Detection System

As described above, conventional systems rely on real-time imagesprocessed using various computer vision or machine learning techniques(e.g., from visual indicators identified via image processing) to detectlanes and/or road boundaries. These techniques are either toocomputationally expensive to accurately perform tasks in real-timeand/or suffer from inaccuracy as a result of shortcuts implemented toreduce computing requirements. As a result, conventional systems fail toprovide the necessary level of accuracy in detecting lanes and/or roadboundaries in real-time by either providing accurate information toolate or inaccurate information unsuitable by an autonomous vehicle tosafely navigate while driving.

In contrast, the present systems provide for an autonomous vehicle thatmay detect lanes and/or road boundaries with increased processingcapability by using a comparatively smaller footprint (e.g., less layersthan conventional approaches) deep neural network (DNN). The DNN may betrained using a variety of different images and ground truth masks—suchas low-resolution full field of view images, higher-resolution region ofinterest (ROI) images, or a combination thereof—in order to increase theaccuracy of the DNN in detecting lanes and road boundaries, especiallyat greater distances. Additionally, because of the architecture of theDNN, the training process for the DNN, and the post-processing of theDNN output, the current systems, when deployed in an autonomous vehicle,may be able to accurately detect lanes and road boundaries—includingthose that are occluded—in real-time and in less than ideal weather orroad conditions.

For example, real-time visual sensor data (e.g., data representative ofimages and/or videos, LIDAR data, RADAR data, etc.) may be received fromsensors (e.g., one or more cameras, one or more LIDAR sensors, one ormore RADAR sensors, etc.) located on an autonomous vehicle. The sensordata may be applied to a machine learning model(s) (e.g., the DNN) thatis trained to identify areas of interest pertaining to road markings,road boundaries, intersections, and/or the like (e.g., raised pavementmarkers, rumble strips, colored lane dividers, sidewalks, cross-walks,turn-offs, etc.) from the sensor data.

More specifically, the machine learning model(s) may be a DNN designedto infer lane and boundary markers and to generate one or moresegmentation masks (e.g., binary and/or multi-class) that may identifywhere in the representations (e.g., image(s)) of the sensor datapotential lanes and road boundaries may be located. In some examples,the segmentation mask(s) may include points denoted by pixels in theimage where lanes and or boundaries may have been determined to belocated by the DNN. In some embodiments, the segmentation mask(s)generated may be a binary mask with a first representation forbackground elements (e.g., elements other than lanes and boundaries) anda second representation for foreground elements (e.g., lanes andboundaries). In other examples, in addition to, or alternative from, thebinary mask, the DNN may be trained to generate a multi-classsegmentation mask, with different classes relating to different lanemarkings and/or boundaries. In such examples, the classes may include afirst class for background elements, a second class for road boundaries,a third class for solid lane markings, a fourth class for dashed lanemarkings, a fifth class for intersections, a sixth class for crosswalks,a seventh class for lane splits, and/or other classes.

The DNN itself may include any number of different layers, although someexamples include fourteen or less layers in order to minimize datastorage requirements and increase processing speeds for the DNN incomparison to conventional approaches. The DNN may include one or moreconvolutional layers, and the convolutional layers may continuously downsample the spatial resolution of the input image (e.g., until the outputlayers, or one or more deconvolutional layers, are reached. Theconvolutional layers may be trained to generate a hierarchicalrepresentation of input images with each layer generating a higher-levelextraction than its preceding layer. As such, the input resolution ateach layer may be decreased, making the DNN capable of processing sensordata (e.g., image data, LIDAR data, RADAR data, etc.) faster thanconventional systems. The DNN may include one or more deconvolutionallayers, which may be the output layer(s) in some examples. Thedeconvolutional layer(s) may up-sample the spatial resolution togenerate an output image of comparatively higher spatial resolution thanthe convolutional layers preceding the deconvolutional layer. The outputof the DNN (e.g., the segmentation mask) may indicate a likelihood of aspatial grid cell (e.g., a pixel) belonging to a certain class of lanesor boundaries.

The DNN may be trained with labeled images using multiple iterationsuntil the value of one or more loss functions of the network are below athreshold loss value. The DNN may perform forward pass computations onthe training images to generate feature extractions of eachtransformation. In some examples, the DNN may extract features ofinterest from the images and predict a probability of the featurescorresponding to a particular boundary class or lane class in the imageson a pixel-by-pixel basis. The loss function(s) may be used to measureerror in the predictions of the DNN using one or more ground truthmasks. In one example, a binary cross entropy function may be used asthe loss function.

Backward pass computations may be performed to recursively computegradients of the loss function with respect to training parameters. Insome examples, weight and biases of the DNN may be used to compute thesegradients. For example, region based weighted loss may be added to theloss function, where the loss function may increasingly penalize loss atfarther distances from the bottom of the image (e.g., representinglocations in a physical environment further from the autonomousvehicle). Advantageously, this may improve detection of lanes andboundaries at farther distances as compared to conventional systemsbecause detecting at further distances may be more finely tuned and thusbetter approximated by the DNN. In some examples, an optimizer may beused to make adjustments to the training parameters (e.g., weights,biases, etc.). In one example, an Adam optimizer may be used, while inothers, stochastic gradient descent, or stochastic gradient descent witha momentum term, may be used to make these adjustments. The trainingprocess (e.g., forward pass computations—backward passcomputations—parameter updates) may be reiterated until the trainedparameters converge to optimum, desired, or acceptable values.

In some non-limiting examples, once the segmentation mask is output bythe DNN, any number of post-processing steps may be performed in orderto ultimately generate lane marking types and curves. In some examples,connected component (CC) labeling may be used. In other examples,directional connected components (DCC) labeling may be used to grouppixels (or points) from the segmentation mask based on the pixel valuesas well as the lane type connectivity in a direction from bottom of theimage to top of the image. By using DCC, as compared to CC labeling, theperspective view (e.g., from the sensor(s) of the vehicle) of the lanemarkings and road boundaries of the driving surface may be takenadvantage of. DCC may also leverage lane appearance type (e.g., based onclasses of the multi-class segmentation mask) when determining whichpixels or points may be connected.

In another non-limiting example, dynamic programming may be used todetermine a set of significant peak points represented by 2D locationsand to determine associated confidence values. For each pair of thesignificant peak points, connectivity may be evaluated, and a set ofpeaks and edges with corresponding connectivity scores may be generated(e.g., based on confidence values). A shortest path algorithm, a longestpath algorithm, and/or all-pairs-shortest path (APSP) algorithms may beused to identify candidate lane edges. In some examples, an additionalcurvature smoothness term may be used when applying an APSP function tocreate a bias toward smooth curves over zig-zag candidate lane edges. Aclustering algorithm may then be used to produce a set of final laneedges by merging sub-paths and similar paths (e.g., identified tocorrespond to candidate lane edges) into one group.

The final lane edges may then be assigned lane types, which may bedetermined relative to a position of the vehicle. Potential lane typesmay include, without limitation, left boundary of the vehicle lane(e.g., ego-lane), right boundary of the vehicle lane, left outerboundary of left-adjacent lane to the vehicle lane, right outer boundaryof right-adjacent lane to the vehicle lane, etc.

In some examples, curve fitting may also be executed in order todetermine final shapes that most accurately reflect a natural curve ofthe lane markings and/or road boundaries. Curve fitting may be performedusing polyline fitting, polynomial fitting, clothoid fitting, and/orother types of curve-fitting algorithms. In some examples, lane curvesmay be determined by resampling segmentation points in the area ofinterest included in the segmentation mask.

Ultimately, data representing the lane markings, lane boundaries, andassociated types may then be compiled and sent to a perception layer, aworld model management layer, a planning layer, a control layer, and/oranother layer of an autonomous driving software stack to aid theautonomous vehicle in navigating the driving surface safely andeffectively.

Now referring to FIG. 1A, FIG. 1A is a data flow diagram illustrating anexample process 100 for detecting lanes and road boundaries, inaccordance with some embodiments of the present disclosure. While thedetection types described with respect to FIG. 1A are lane and roadboundary detection, this is not intended to be limiting, and is used forexample purposes only.

The process 100 for lane and road boundary detection may includegenerating and/or receiving sensor data 102 from one or more sensors ofthe autonomous vehicle 800. The sensor data 102 may include sensor datafrom any of the sensors of the vehicle 800 (and/or other vehicles orobjects, such as robotic devices, VR systems, AR systems, etc., in someexamples). With reference to FIGS. 8A-8C, the sensor data 102 mayinclude the data generated by, for example and without limitation,global navigation satellite systems (GNSS) sensor(s) 858 (e.g., GlobalPositioning System sensor(s)), RADAR sensor(s) 860, ultrasonic sensor(s)862, LIDAR sensor(s) 864, inertial measurement unit (IMU) sensor(s) 866(e.g., accelerometer(s), gyroscope(s), magnetic compass(es),magnetometer(s), etc.), microphone(s) 896, stereo camera(s) 868,wide-view camera(s) 870 (e.g., fisheye cameras), infrared camera(s) 872,surround camera(s) 874 (e.g., 360 degree cameras), long-range and/ormid-range camera(s) 898, speed sensor(s) 844 (e.g., for measuring thespeed of the vehicle 800), vibration sensor(s) 842, steering sensor(s)840, brake sensor(s) (e.g., as part of the brake sensor system 846),and/or other sensor types.

In some examples, the sensor data 102 may include the sensor datagenerated by one or more forward-facing cameras (e.g., a center ornear-center mounted camera(s)), such as a wide-view camera 870, asurround camera 874, a stereo camera 868, and/or a long-range ormid-range camera 898. This sensor data may be useful for computer visionand/or perception when navigating—e.g., within a lane, through a lanechange, through a turn, through an intersection, etc.—because aforward-facing camera may include a field of view (e.g., the field ofview of the forward-facing stereo camera 868 and/or the wide-view camera870 of FIG. 8B) that includes both a current lane of travel of thevehicle 800, adjacent lane(s) of travel of the vehicle 800, and/orboundaries of the driving surface. In some examples, more than onecamera or other sensor (e.g., LIDAR sensor, RADAR sensor, etc.) may beused to incorporate multiple fields of view (e.g., the fields of view ofthe long-range cameras 898, the forward-facing stereo camera 868, and/orthe forward facing wide-view camera 870 of FIG. 8B).

In any example, the sensor data 102 may include image data representingan image(s), image data representing a video (e.g., snapshots of video),and/or sensor data representing fields of view of sensors (e.g., LIDARsensor(s) 864, RADAR sensor(s) 860, etc.). In some examples, the sensordata 102 may be input into the machine learning model(s) 108 and used bythe machine learning model(s) 108 to compute segmentation mask(s) 110.In some other examples, the sensor data 102 may be provided as input tothe sensor data pre-processor 104 to generate pre-processed sensor data106. The pre-processed sensor data 106 may then be input into themachine learning model(s) 108 as input data.

Many types of images or formats may be used as inputs, for example,compressed images such as in Joint Photographic Experts Group (JPEG) orLuminance/Chrominance (YUV) formats, compressed images as framesstemming from a compressed video format such as H.264/Advanced VideoCoding (AVC) or H.265/High Efficiency Video Coding (HEVC), raw imagessuch as originating from Red Clear Blue (RCCB), Red Clear (RCCC) orother type of imaging sensor. It is noted that different formats and/orresolutions could be used training the machine learning model(s) 108than for inferencing (e.g., during deployment of the machine learningmodel(s) 108 in the autonomous vehicle 800).

The sensor data pre-processor 104 may use sensor data representative ofone or more images (or other data representations) and load the sensordata into memory in the form of a multi-dimensional array/matrix(alternatively referred to as tensor, or more specifically an inputtensor, in some examples). The array size may be computed and/orrepresented as W×H×C, where W stands for the image width in pixels, Hstands for the height in pixels and C stands for the number of colorchannels. Without loss of generality, other types and orderings of inputimage components are also possible. Additionally, the batch size B maybe used as a dimension (e.g., an additional fourth dimension) whenbatching is used. Batching may be used for training and/or forinference. Thus, the input tensor may represent an array of dimensionW×H×C×B. Any ordering of the dimensions may be possible, which maydepend on the particular hardware and software used to implement thesensor data pre-processor 104. This ordering may be chosen to maximizetraining and/or inference performance of the machine learning model(s)108.

A pre-processing image pipeline may be employed by the sensor datapre-processor 104 to process a raw image(s) acquired by a sensor(s) andincluded in the sensor data 102 to produce pre-processed sensor data 106which may represent an input image(s) to the input layer(s) (e.g.,convolutional streams(s) 132 of FIG. 1B) of the machine learningmodel(s) 108. An example of a suitable pre-processing image pipeline mayuse a raw RCCB Bayer (e.g., 1-channel) type of image from the sensor andconvert that image to a RCB (e.g., 3-channel) planar image stored inFixed Precision (e.g., 16-bit-per-channel) format. The pre-processingimage pipeline may include decompanding, noise reduction, demosaicing,white balancing, histogram computing, and/or adaptive global tonemapping (e.g., in that order, or in an alternative order).

Where noise reduction is employed by the sensor data pre-processor 104,it may include bilateral denoising in the Bayer domain. Wheredemosaicing is employed by the sensor data pre-processor 104, it mayinclude bilinear interpolation. Where histogram computing is employed bythe sensor data pre-processor 104, it may involve computing a histogramfor the C channel, and may be merged with the decompanding or noisereduction in some examples. Where adaptive global tone mapping isemployed by the sensor data pre-processor 104, it may include performingan adaptive gamma-log transform. This may include calculating ahistogram, getting a mid-tone level, and/or estimating a maximumluminance with the mid-tone level.

The machine learning model(s) 108 may use as input one or more images(or other data representations) represented by the sensor data 102 togenerate one or more segmentation masks 110 as output. In a non-limitingexample, the machine learning model(s) 108 may take as input an image(s)represented by the pre-processed sensor data 106 (alternatively referredto herein as “sensor data 106”) to generate a segmentation mask(s) 110.Although examples are described herein with respect to using neuralnetworks, and specifically convolutional neural networks, as the machinelearning model(s) 108 (e.g., with respect to FIGS. 1B, 1C, 3A, 3C, and7C), this is not intended to be limiting. For example, and withoutlimitation, the machine learning model(s) 108 described herein mayinclude any type of machine learning model, such as a machine learningmodel(s) using linear regression, logistic regression, decision trees,support vector machines (SVM), Naïve Bayes, k-nearest neighbor (Knn), Kmeans clustering, random forest, dimensionality reduction algorithms,gradient boosting algorithms, neural networks (e.g., auto-encoders,convolutional, recurrent, perceptrons, Long/Short Term Memory (LSTM),Hopfield, Boltzmann, deep belief, deconvolutional, generativeadversarial, liquid state machine, etc.), and/or other types of machinelearning models.

The segmentation mask(s) 110 output by the machine learning model(s) 108may represent portions of the input image(s) determined to correspond tolane markings or road boundaries of a driving surface of the vehicle800. The machine learning model(s) 108 may include one or more neuralnetworks trained to generate the segmentation mask(s) 110 as output thatidentifies where in the image(s) potential lanes and boundaries may belocated. In a non-limiting example, the segmentation mask(s) 110 mayfurther represent confidence scores corresponding to a probability ofeach of the portions of the mask corresponding to potential lanes and/orroad boundaries. In addition, in some examples, the segmentation mask(s)110 may further represent confidence scores corresponding toprobabilities of each of the portions of the mask corresponding to acertain class of lane marking or road boundary (e.g., a lane markingtype and/or a road boundary type).

In some examples, the segmentation mask(s) 110 may include points (e.g.,pixels) in the image where lanes and or road boundaries are determinedto be located by the machine learning model(s) 108. In some examples,the segmentation mask(s) 110 generated may include one or more binarymasks (e.g., binary mask head 334 of FIG. 3C) with a firstrepresentation for background elements (e.g., elements other than lanesand road boundaries) and a second representation for foreground elements(e.g., lanes and road boundaries). The binary mask may be output by themachine learning model(s) 108 as pixel values of 0 or 1 (for black orwhite), may include other pixel values, or may include a range of valuesthat are interpreted as 0 or 1 (e.g., 0 to 0.49 is interpreted as 0, and0.5 to 1 is interpreted as 1). A resulting visualization (e.g., theillustration of the segmentation mask 110 in FIG. 1A-1B) may be a blackand white version of the image. Although the lines, boundaries, and/orother features are included as black in the illustration, and thebackground elements are white, this is not intended to be limiting. Forexample, the background elements may be black and the foreground may bewhite, or other colors may be used.

In other examples, the machine learning model(s) 108 may be trained togenerate one or more multi-class segmentation masks (e.g., multi-classmask head 332 of FIG. 3C) as the segmentation mask(s) 110, withdifferent classes relating to different lane markings and/or boundaries.In such examples, the classes may include a first class for backgroundelements, a second class for road boundaries, a third class for solidlane markings, a fourth class for dashed lane markings, a fifth classfor intersections, a sixth class for crosswalks, a seventh class forlane splits, and/or additional or alternative classes. A resultingvisualization (e.g., the illustration of the multi-class mask head 332of FIG. 3C) may include a first pixel value for background elements, asecond pixel value for road boundaries (e.g., a pixel valuecorresponding to red), a third pixel value for solid lane markings(e.g., a pixel value corresponding to green), and so on.

The segmentation mask(s) 110 output by the machine learning model(s) 108may undergo post-processing. For example, the segmentation mask(s) 110may undergo resampling 112. The resampling 112 may include extractingpoints (e.g., pixels) from the segmentation mask(s) 110 where the pointsmay correspond to lanes (e.g., lane markings) and/or road boundaries asdetermined by the machine learning model(s) 108. The resampling 112 mayinclude grouping pixels in the segmentation mask(s) 110 into lanecomponents for each area of interest (e.g., each detected lane markingor road boundary).

In some non-limiting examples, connected components (CC) labeling may beused to group the points. In other non-limiting examples, directionalconnected components (DCC) 114 labeling may be used to group points fromthe segmentation mask(s) 110 based on the pixel values and/or lane typeconnectivity (e.g., white dashed line, white solid line, yellow dashedline, yellow solid line, etc.). As compared to CC labeling, DCC 114 mayscan the image(s) from bottom to top, thereby taking advantage of theperspective view (e.g., from the sensor(s) of the vehicle 800) of thelane markings and/or road boundaries of the driving surface. In suchexamples, DCC 114 may compare or examine a bottom neighbor or bottomadjacent point of a given point to increment or determine whether thegiven point is from the same lane marking type and thus should beconnected. In some examples, DCC 114 may leverage lane appearance type(e.g., based on classes of the multi-class segmentation mask) whendetermining which points (e.g., pixels) should be connected. Forexample, a given point may not be grouped with its corresponding bottomneighbor point if the bottom neighbor point belongs to a different laneappearance type.

In another non-limiting example, dynamic programming 116 may be used aspart of the resampling 112 of the process 100. Dynamic programming 116may include determining a set of significant peak points (e.g., pixels)represented by 2D locations and associated confidence values for eacharea of interest (e.g., each detected lane and/or road boundary). One ormore methods, such as but not limited to those described herein, may beused to determine the set of significant peak points. In a non-limitingexample, the set of significant points may be determined by performingnon-maxima suppression after Gaussian smoothing of the points (e.g.,pixels) in the area(s) of interest of the segmentation mask(s) 110. Forall pairs of the peak points, connectivity may be evaluated, and a setof peak points and edges with corresponding connectivity scores may begenerated (e.g., based on confidence values). In some examples, theconnectivity may be computed as a sum of confidence values of all pointsin the area of interest between the pair of peak points. In anotherexample, a fixed number of equally sampled points (e.g., pixels) betweenthe pair may be used to generate the connectivity scores. In yet afurther example, confidence values for all peak points may be fed into arobustifier function, such as an exponential or Cauchy function. Therobustifier function may be used to generate connectivity scores foreach pair of peak points. As compared to conventional connectedcomponents (CC) labeling, these functions detect connection sensitivityat even a weak connection between peak points). In some examples, classlabels, such as those in the multi-class segmentation mask(s) may beused to further refine connectivity between pixels.

Dynamic programing 116 may further include using a shortest pathalgorithm, a longest path algorithm, and/or all-pairs-shortest path(APSP) algorithm to identify candidate lane edges. In some examples,connectivity of the peak points (e.g., pixels) may be formulated interms of cost. In such examples, lane edges may be identified using ashortest path algorithm. In another example, connectivity of the peakpoints may be formulated in terms of likelihood of connection. In suchan example, the lane edges may be identified using a longest pathalgorithm. Although examples are described herein with respect to usinga shortest path algorithm, a longest path algorithm, and/or an APSPalgorithm to determine lane edges as part of dynamic programming 116,this is not intended to be limiting. For example, and withoutlimitation, the dynamic programming 116 described herein may include anytype and/or combination of algorithms to identify candidate lane edges.

In some non-limiting examples, the dynamic programming 116 may use anadditional curvature smoothness term when identifying lane edges tocreate a bias toward smooth curves over zig-zag candidate lane edges. Inone example, the preference may be adjusted using a control parameter inan optimization algorithm.

In any example, a clustering algorithm may be used to produce a set offinal lane edges by merging sub-paths and similar paths (e.g.,identified to correspond to candidate lane edges) into one group. Insome examples, topological or spatial clustering algorithms may beapplied sequentially and/or in tandem. Topological clustering algorithmsmay be used to merge two paths if one path is a sub-part of the other.Additionally or alternatively, if two paths share common pairs of peakpoints, the path with a lower likelihood or higher cost may be mergedwith the one with the higher likelihood or lower cost. Spatialclustering algorithms may merge paths based on similarities betweengeometries of paths. For example, a spatial clustering algorithm maymerge a path with lower likelihood or higher cost to a path with higherlikelihood or lower cost (e.g., when the two paths are determined to begeometrically similar to one another).

The final lane edges derived by resampling 112 may then undergo laneassignment 118 to be assigned lane types and/or road boundary types. Insome examples, the lane types and/or road boundary types may bedetermined relative to a position of the vehicle 800. For example, lanetypes (e.g., lane-marking types) may include a left boundary of thevehicle lane (e.g., the ego-lane), right boundary of the vehicle lane,left outer boundary of left-adjacent lane to the vehicle lane, rightouter boundary of right-adjacent lane to the vehicle lane, and/or othertypes.

In some examples, it may be assumed that the principal axis of thesensor that generated the sensor data 102 is approximately aligned withthe roll axis of the vehicle 800 (e.g., a longitudinal axis). In suchexamples, the sensor may actually be aligned with the roll axis, inothers, the sensor may be positioned within a threshold distance fromthe roll axis that the sensor data 102 is useable, and/or the sensordata 102 may be transformed (e.g., shifted) based on calibration data ofthe sensor (e.g., based on a distance from the roll axis). The lanemarking types and/or road boundary types may then be determined based onthis assumption. For example, a lane marking to the right of a verticalcenterline (e.g., extending from bottom to top) of an image (e.g.,representing the sensor data 102) may be determined to be the rightboundary of the vehicle lane, the next lane marking to the right may bethe right outer boundary of the right-adjacent lane of the vehicle 800,and so on. Similarly, for the left of the vertical centerline of theimage, a lane marking to the left of the vertical centerline may bedetermined to be the left boundary of the vehicle lane, the next lanemarking to the left may be the left outer boundary of the left-adjacentlane of the vehicle 800, and so on.

More specifically, for each lane edge, the bottom of the edge may beextended to meet the bottom of the corresponding image. As such, foreach lane edge, the intersection of the extended lane edge with thebottom of the image may be determined in terms of column difference (ordistance) from the intersection with the bottom of the image to thevertical centerline of the image (e.g., the principal axis). The laneedge associated with the minimum positive column difference (e.g., theminimum column difference to the right of the principal axis) may beidentified as right boundary of the vehicle lane. The lane edge with thesecond smallest positive column difference (e.g., the second smallestcolumn difference to the right of the principal axis) may be identifiedas the right boundary of the right-adjacent lane to the vehicle lane.Similarly, the lane edge associated with the minimum negative columndifference (e.g., the minimum column difference to the left of theprincipal axis) may be identified as left boundary of the vehicle lane,and the lane edge with the second smallest negative column difference(e.g., the second smallest column difference to the left of theprincipal axis) may be identified as the left boundary of theleft-adjacent lane to the vehicle lane. In some examples, this labelingmay extend to any number of lanes and/or road boundaries. In otherexamples, only a certain number of lanes and/or road boundaries may belabeled, and any remaining lanes and/or road boundaries may be labeledas undefined. In such examples, one or more of the remaining lanesand/or road boundaries that are identified may be removed and/or notincluded in any further processing by the vehicle 800.

Curve fitting 120 may also be implemented in order to determine finalshapes of the potential lanes and/or boundaries identified that mostaccurately reflect a natural curve of the lane markings and/orboundaries. Curve fitting 120 may be performed using polyline fitting,polynomial fitting, clothoid fitting, and/or other types ofcurve-fitting algorithms. In examples where clothoid fitting is used,curve fitting 120 may include tuning the number of clothoids in theclothoid fitting algorithm to fit the curve of the driving surface ofvehicle 800. In some examples, the curve fitting 120 may be performedusing the lane edges identified from resampling 112 and/or laneassignment 118. In other examples, curve fitting 120 may be performed byresampling points (e.g., segmentation points) in the area(s) of interestincluded in the segmentation mask(s) 110 (as indicated by the dashedline in FIG. 1A).

The output of the resampling 112, the lane assignment 118, and/or thecurve fitting 120 may then be used (e.g., after compiling) to generatedata representative of lane labels (or assignments) and lane curves 122,respectively. Ultimately, data representing the lane markings, laneboundaries, and/or associated label types may then be compiled and sentto one or more layers of the autonomous driving software stack, such asa world model management layer, a perception layer, a planning layer, acontrol layer and/or another layer. The autonomous driving softwarestack may thus use the data to aid in navigating the vehicle 800 throughthe driving surface within the physical environment.

Now referring to FIG. 1B, FIG. 1B is an illustration of an examplemachine learning model(s) 108A, in accordance with some embodiments ofthe present disclosure. The machine learning model(s) 108A of FIG. 1Bmay be one example of a machine learning model(s) 108 that may be usedin the process 100. However, the machine learning model(s) 108A of FIG.1B is not intended to be limiting, and the machine learning model(s) 108may include additional and/or different machine learning models than themachine learning model(s) 108A of FIG. 1B. The machine learning model(s)108A may include or be referred to as a convolutional neural network andthus may alternatively be referred to herein as convolutional neuralnetwork 108A or convolutional network 108A.

The convolutional network 108 may use the sensor data 102 and/or thepre-processed sensor data 106 as an input. For example, theconvolutional network 108A may use the sensor data 130—as represented bythe sensor data 130A-130C—as an input. The sensor data 130 may includeimages representing image data generated by one or more cameras (e.g.,one or more of the cameras described herein with respect to FIGS.8A-8C). For example, the sensor data 130A-130C may include image datarepresentative of a field of view of the camera(s). More specifically,the sensor data 130A-130C may include individual images generated by thecamera(s), where image data representative of one or more of theindividual images may be input into the convolutional network 108 ateach iteration of the convolutional network 108.

The sensor data 102 and/or pre-processed sensor data 106 may be inputinto a convolutional layer(s) 132 of the convolutional network 108(e.g., convolutional layer 134A). The convolutional stream 132 mayinclude any number of layers 134, such as the layers 134A-134C. One ormore of the layers 134 may include an input layer. The input layer mayhold values associated with the sensor data 102 and/or pre-processedsensor data 106. For example, when the sensor data 102 is an image(s),the input layer may hold values representative of the raw pixel valuesof the image(s) as a volume (e.g., a width, W, a height, H, and colorchannels, C (e.g., RGB), such as 32×32×3), and/or a batch size, B.

One or more layers 134 may include convolutional layers. Theconvolutional layers may compute the output of neurons that areconnected to local regions in an input layer (e.g., the input layer),each neuron computing a dot product between their weights and a smallregion they are connected to in the input volume. A result of aconvolutional layer may be another volume, with one of the dimensionsbased on the number of filters applied (e.g., the width, the height, andthe number of filters, such as 32×32×12, if 12 were the number offilters).

One or more of the layers 134 may include a rectified linear unit (ReLU)layer. The ReLU layer(s) may apply an elementwise activation function,such as the max (0, x), thresholding at zero, for example. The resultingvolume of a ReLU layer may be the same as the volume of the input of theReLU layer.

One or more of the layers 134 may include a pooling layer. The poolinglayer may perform a down-sampling operation along the spatial dimensions(e.g., the height and the width), which may result in a smaller volumethan the input of the pooling layer (e.g., 16×16×12 from the 32×32×12input volume). In some examples, the convolutional network 108A may notinclude any pooling layers. In such examples, strided convolution layersmay be used in place of pooling layers.

One or more of the layers 134 may include a fully connected layer. Eachneuron in the fully connected layer(s) may be connected to each of theneurons in the previous volume. The fully connected layer may computeclass scores, and the resulting volume may be 1×1×number of classes. Insome examples, the convolutional stream(s) 132 may include a fullyconnected layer, while in other examples, the fully connected layer ofthe convolutional network 108 may be the fully connected layer separatefrom the convolutional streams(s) 132.

Although input layers, convolutional layers, pooling layers, ReLUlayers, and fully connected layers are discussed herein with respect tothe convolutional layer(s) 134, this is not intended to be limiting. Forexample, additional or alternative layers 134 may be used in theconvolutional stream(s) 132, such as normalization layers, SoftMaxlayers, and/or other layer types.

The output of the convolutional stream 132 and/or the convolutionallayer(s) 134 may be an input to deconvolutional layer(s) 136. Althoughreferred to as deconvolutional layer(s) 136, this may be misleading andis not intended to be limiting. For example, the deconvolutionallayer(s) 136 may alternatively be referred to as transposedconvolutional layers or fractionally strided convolutional layers. Thedeconvolutional layer(s) 136 may be used to perform up-sampling on theoutput of a prior layer (e.g., a layer 134 of the convolutionalstream(s) 132 and/or an output of another deconvolutional layer). Forexample, the deconvolutional layer(s) 136 may be used to up-sample to aspatial resolution that is equal to the spatial resolution of the inputimages (e.g., the images 130) to the convolutional network 108A.

Different orders and numbers of the layers 134 and/or 136 of theconvolutional network 108A may be used depending on the embodiment. Forexample, for a first vehicle, there may be a first order and number oflayers 134 and/or 136, whereas there may be a different order and numberof layers 134 and/or 136 for a second vehicle; for a first camera, theremay be a different order and number of layers 134 and/or 136 than theorder and number of layers for a second camera. In other words, theorder and number of layers 134 and/or 136 of the convolutional network108A, the convolutional stream 132, and/or the deconvolutional layer(s)136 is not limited to any one architecture.

In addition, some of the layers 134 may include parameters (e.g.,weights and/or biases), such as the layers of the convolutional stream132 and/or the deconvolutional layer(s) 136, while others may not, suchas the ReLU layers and pooling layers, for example. In some examples,the parameters may be learned by the convolutional stream 132 and/or themachine learning model(s) 108A during training. Further, some of thelayers 134 and/or 136 may include additional hyper-parameters (e.g.,learning rate, stride, epochs, kernel size, number of filters, type ofpooling for pooling layers, etc.), such as the convolutional layers 134,the deconvolutional layer(s) 136, and the pooling layers (as part of theconvolutional stream(s) 132), while other layers 142 may not, such asthe ReLU layers. Various activation functions may be used, including butnot limited to, ReLU, leaky ReLU, sigmoid, hyperbolic tangent (tan h),exponential linear unit (ELU), etc. The parameters, hyper-parameters,and/or activation functions are not to be limited and may differdepending on the embodiment.

Now referring to FIG. 1C, FIG. 1C is an illustration of another examplemachine learning model(s) 108B in accordance with some embodiments ofthe present disclosure. In some examples, the convolutional network 108Amay include any number of different layers, although some examplesinclude fourteen or less layers in order to minimize data storagerequirements and to increase processing speeds for the convolutionalnetwork 108B. The convolutional layers 134 may continuously down samplethe spatial resolution of the input image until the output layers arereached (e.g., down-sampling from a 480×252 input spatial resolution atlayer 134A to 240×126 as output of layer 134A, down-sampling from240×126 input spatial resolution at layer 134E to 120×63 as output oflayer 134E, etc.). The convolutional stream(s) 132 may be trained togenerate a hierarchical representation of the input image(s) receivedfrom the sensor data 102 and/or pre-processed sensor data 106 (e.g., theimages 130) with each layer generating a higher-level extraction thanits preceding layer. In other words, as can be seen in FIG. 1C, theinput resolution across the convolutional layers 134A-134M (and/or anyadditional or alternative layers) may be decreased, allowing theconvolutional network 108A to be capable of processing images fasterthan conventional systems.

The output layer(s) 136, similar to in FIG. 1B, may be a deconvolutionlayer(s) that up samples the spatial resolution to generate an outputimage of comparatively higher spatial resolution than the convolutionallayers preceding the deconvolution layer. The output of theconvolutional network 108B (e.g., the segmentation mask(s) 110,alternatively referred to as coverage map(s)) may indicate a likelihoodof a spatial grid cell belonging to a certain class of lanes orboundaries.

In some examples, the machine learning model(s) 108 (e.g., a neuralnetwork(s)) may be trained with labeled images using multiple iterationsuntil the value of a loss function(s) of the machine learning model(s)108 is below a threshold loss value. For example, the machine learningmodel(s) 108 may perform forward pass computations on therepresentations (e.g., image(s)) of the sensor data 102 and/orpre-processed sensor data 106 to generate feature extractions. In someexamples, the machine learning model(s) 108 may extract features ofinterest from the image(s) and predict probability of boundary classesand/or lane classes in the images on a pixel-by-pixel basis. The lossfunction(s) may be used to measure error in the predictions of themachine learning model(s) 108 using ground truth masks, as described inmore detail herein with respect to at least FIGS. 3A, 4A-4B, 5A-5B, and6A-6E.

In some examples, a binary cross entropy function may be used as a lossfunction. Backward pass computations may be performed to recursivelycompute gradients of the loss function with respect to trainingparameters. In some examples, weights and biases of the machine learningmodel(s) 108 may be used to compute these gradients. For example, regionbased weighted loss may be added to the loss function, where the lossfunction may increasingly penalize loss at distances further from abottom of the image(s) (e.g., distances further from the vehicle 800).By using region based weighted loss, detections of lanes and/orboundaries at further distances may be improved as compared toconventional systems. For example, the region based weighted lossfunction may result in back-propagation of more error at furtherdistances during training, thereby reducing the error in predictions bythe machine learning model(s) 108 at further distances during deploymentof the machine learning model(s) 108.

In some examples, an optimizer may be used to make adjustments to thetraining parameters (e.g., weights, biases, etc.). In one example, anAdam optimizer may be used, while in other examples, stochastic gradientdescent, or stochastic gradient descent with a momentum term, may beused. The training process may be reiterated until the trainedparameters converge to optimum, desired, and/or acceptable values.

Now referring to FIG. 2 , each block of method 200, described herein,may comprise a computing process that may be performed using anycombination of hardware, firmware, and/or software. For instance,various functions may be carried out by a processor executinginstructions stored in memory. The methods may also be embodied ascomputer-usable instructions stored on computer storage media. Themethods may be provided by a standalone application, a service or hostedservice (standalone or in combination with another hosted service), or aplug-in to another product, to name a few. In addition, method 200 isdescribed, by way of example, with respect to the vehicle 800 and theprocess 100. However, these methods may additionally or alternatively beexecuted by any one system, or any combination of systems, including,but not limited to, those described herein.

FIG. 2 is a flow diagram showing a method 200 for detecting lanes and/orroad boundaries, in accordance with some embodiments of the presentdisclosure. The method 200, at block B202, includes receiving sensordata. For example, sensor data 102 may be generated and/or captured byone or more sensors (e.g., cameras, LIDAR sensors, RADAR sensors, etc.)of the vehicle 800 and may be received after generation, capture, and/orpre-processing (e.g., as the sensor data 106). The sensor data 102and/or 106 may include sensor data (e.g., image data) representative offield(s) of view of one or more sensors. In examples where the sensordata that is received is the sensor data 106, the sensor data 106 may begenerated by the sensor data pre-processor 104.

The method 200, at block B204, includes applying the sensor data to aneural network(s). For example, the sensor data 102 and/or 106representative of field(s) of view of the one or more sensors of thevehicle 800 may be applied to the machine learning model(s) 108.

The method 200, at block B206, includes computing, by the neuralnetwork(s), segmentation mask(s). For example, the machine learningmodel(s) 108 may compute the segmentation mask(s) 110 based at least inpart on sensor data 102 and/or the pre-processed sensor data 106. Thesegmentation mask may include data representative of portions of thesensor data 102, 106 and/or representations thereof (e.g., images)determined to correspond to lane markings and/or boundaries of a drivingsurface of the vehicle 800.

The method 200, at block B208, includes assigning lane marking types.For example, lane assignment 118 may be performed to assign lane markingtypes to each of the lane markings and/or boundary markings based atleast in part on the segmentation mask(s) 110.

The method 200, at block B210, includes performing curve fitting on thelane markings. For examples, curve fitting 120 may be performed on thelane markings, the boundary markings, and/or on segmentation points ofthe segmentation mask(s) 110 to generate lane markings and/or boundariesrepresentative of the lane marking types. As described herein, the curvefitting 120 may be based at least in part on the segmentation mask(s)110.

The method 200, at block B212, includes sending data representative ofthe lane boundaries to a component of the vehicle at least for use bythe vehicle in navigating the driving surface. For example, the datarepresentative of the lane boundaries and/or road boundaries, such asthe lane curves and labels 122, may be sent to a planning layer 124, acontrol layer, a perception layer, a world model management layer, anobstacle avoidance layer, and/or another layer of an autonomous drivingsoftware stack of the vehicle 800 for use by the vehicle 800 innavigating the driving surface in the physical environment. As such, thevehicle 800 may use the lane curves and labels 122 to perform variousdriving maneuvers, such as lane keeping, lane changing, turns, lanemerges, lane splits, stopping, starting, slowing down, etc.

Training Machine Learning Model(s)

As described above, conventional systems rely on processing images usingvarious computer vision or machine learning techniques (e.g., fromvisual indicators identified via image processing) to detect lanesand/or road boundaries. However, these conventional processes are eithertoo computationally expensive to perform accurately in real-time and/orsuffer from inaccuracy as a result of shortcuts implemented to reducecomputing requirements for real-time deployment. As a result,conventional systems may fail to provide the necessary level of accuracyin detecting lanes and/or road boundaries in real-time.

In contrast, the present system provides for lane and/or road boundarydetection in real-time at practically acceptable accuracy levels. Toaccomplish this, a comparatively small footprint (e.g., less layers thanconventional approaches) machine learning model(s) (e.g., a DNN) may beused as part of the lane and road boundary detection system (asdescribed herein). In addition, the machine learning model(s) may betrained using full field of view images, region of interest (ROI) images(e.g., cropped images), and/or a combination thereof in order toincrease the accuracy of the DNN in detecting lanes and road boundaries,especially at greater distances. The full field of view images and/orthe cropped images may also be down-sampled to lower spatial resolutionsprior to being input into the machine learning model(s) in order tofurther increase processing time (e.g., less spatial resolution mayresult in less nodes and thus less data to process by the model(s)). Asa result, the training process, and generation of the training and/orground truth data, may contribute to increasing the processing speedsfor the current system such that lane and road boundary detection mayhappen in real-time at an acceptable level of accuracy for safeoperation of an autonomous vehicle (or other object).

With reference to training the machine learning model(s) (e.g., theDNN), in some examples, the machine learning model(s) may be trainedusing original images and transformed or augmented versions of theoriginal images. However, in order to accurately train the machinelearning model(s) with original and transformed images, ground truthinformation (e.g., annotations, masks, labels, etc.) associated with theimages may also need to undergo similar transformations or augmentationstheir corresponding images. In some examples, the images may undergo aspatial transformation (e.g., left flip, right flip, zoom in, a zoomout, a random translation, etc.), a color transformation, and/or anothertransformation type, and the ground truth information (e.g.,annotations, masks, labels, etc.) may undergo correspondingtransformations.

For example, an original image may be associated with vertices of apolygon (e.g., an annotation of vertices rendered with respect to theoriginal image) representing a lane or a boundary. The original imagemay undergo a transformation to generate a transformed image. Thevertices of the polygon may similarly undergo a transformation (e.g.,change in location) to generate a transformed polygon based on thetransformation of the image, such that the transformed vertices may berendered in a corresponding location with respect to the transformedimage. A masked image may then be generated by rendering a mask over theportion of the transformed image including the transformed polygon. Themachine learning model(s) may be trained using the masked image asground truth data. This may enable the machine learning model(s) tolearn features from a virtually larger set of images, while avoidingoverfitting of the model(s) to the training data.

In another non-limiting example, the machine learning model(s) may betrained using down-sampled and/or ROI images. The ROI images mayrepresent a cropped image (e.g., center crop, right crop, left crop,half size, etc.). The cropped image may include a portion of a polygon(e.g., a polygon from the annotations of the original image)representing a lane or boundary outside, and the polygon may extendbeyond the cropped portion of the image. In such a case, a ground truthmask may be generated by masking the cropped portion of the polygon inthe cropped image using a first mask, and by masking out the portion ofthe polygon that is outside of the cropped portion of the image using asecond mask. As a result, only the portion of the polygon that is withinthe ROI (e.g., the cropped region) may be masked for the purposes oftraining the machine learning model(s).

In examples where both the down-sampled and ROI images are used, bothimages may be applied to the machine learning model(s) at the same time(e.g., as a batch or mini-batch). This may allow the machine learningmodel(s) to learn features from a more accurate set of points (e.g.,pixels) of the image while avoiding learning features from outside ofthe ROI. In addition, by using down-sampled and/or ROI images, themachine learning model(s) may learn from images with different fields ofview, thus increasing the accuracy of the predictions by the machinelearning model(s) even with respect to a single image from a singlefield of view once deployed for lane and boundary detection in anautonomous vehicle. As such, in some examples, the machine learningmodel(s) may be trained using two or more image types (e.g., an originalimage, a down-sampled image, an ROI image, etc.), but may only require asingle image type for lane and boundary detection once fully trained(e.g., deployed). However, this is not intended to be limiting, and evenin deployment, the machine learning model(s) may use two or more imagetypes in a batch or mini-batch. In any example, by training the machinelearning model(s) according to these processes, the processing powerrequired for real-time deployment may be reduced because the machinelearning model(s) may learn to generate the segmentation mask(s) withimages of lower input resolutions than those of conventional systems.

In some examples, labels or annotations (e.g., polygons) may be created(e.g., rendered, drawn, etc.) for lane merges and/or lane splits in aplurality of training images (e.g., ground truth images). The labels orannotations may be tagged (or labeled) as lane merge lane marking orlane split lane marking types, and a tip, top, or point, of the labelsor annotations (e.g., a side of the polygon corresponding to the pointof lane split or lane merge) may be separately labeled to identify thepoint of lane split or the point of lane merge on the labels orannotations. The training images may then be used as ground truth datato train the machine learning model(s) to learn to identify (e.g., aspart of a segmentation mask) lane merges and lane splits. In exampleswhere the segmentation mask(s) is a multi-class mask, a prediction of aclass of lane merge or class of lane split may also be output by themachine learning model(s).

In other examples, the training images may include images ofintersections, crosswalks, or a combination thereof. The ground truthinformation (e.g., annotations, labels, masks, etc.) for these imagesmay include separate labels, annotations, or masks (e.g., polygons) forthe crosswalk and/or the intersection. For example, where the imageincludes a crosswalk at an intersection (e.g., a crosswalk divingintersection lines), the crosswalk may include separate labels from theintersection, such that during training the machine learning model(s)learns to differentiate between the portion of the image corresponds tothe crosswalk and the portion of the image that corresponds to theintersection. In examples where the segmentation mask(s) is amulti-class mask, a prediction of a class of crosswalk or class ofintersection may be output by the machine learning model(s). Ultimately,the autonomous vehicle implementing the examples where the segmentationmask(s) is a multi-class mask, a prediction of a class of lane merge orclass of lane split may also be output by the machine learning model(s)may learn to behave differently where there is a crosswalk, anintersection, or a combination thereof.

Now referring to FIG. 3A, FIG. 3A is a data flow diagram illustrating anexample process 300 for training a machine learning model(s) to detectlanes and road boundaries, in accordance with some embodiments of thepresent disclosure. Although the detections described with respect toFIG. 3A relate to lanes and road boundaries, this is not intended to belimiting, and the detections may also be for crosswalks, intersection,lane splits, lane merges, parking lines, lines within or around astructure (e.g., directional lines within a building for navigating thebuilding, such as for robots, lines of a field or other space for VRapplications, etc.), and/or other detections without departing from thescope of the present disclosure.

As described herein, the machine learning model(s) 108 may be trainedusing original images, down-sampled images, up-sampled images, region ofinterest (ROI) images, and/or a combination thereof. One or more ofthese image types may be included within input images 302 used fortraining the machine learning model(s) 108. The input images 302 may beimages captured by one or more sensors (e.g., cameras) of variousvehicles (e.g., the vehicle 800), and/or may be images captured fromwithin a virtual environment used for testing and/or generating trainingimages. In some examples, the input images 302 may be images from a datastore or repository of training images (e.g., images of driving surfacesincluding lane markings, boundary markings, crosswalk markings,intersection markings, lane split markings, lane merge markings, etc.).The machine learning model(s) 108 may be trained using both the inputimages 302 corresponding labels 308 (e.g., as ground truth data) todetect lanes and/or boundaries on driving surfaces. The input images 302may have corresponding labels 308, which may include annotations,labels, masks, and/or the like. The labels 308 may be generated within adrawing program (e.g., an annotation program), a computer aided design(CAD) program, a labeling program, another type of program suitable forgenerating the labels 308, and/or may be hand drawn, in some examples.In any example, the labels 308 may be synthetically produced (e.g.,generated from computer models or renderings), real produced (e.g.,designed and produced from real-world data), machine-automated (e.g.,using feature analysis and learning to extract features from data andthen generate labels), human annotated (e.g., labeler, or annotationexpert, defines the location of the labels), and/or a combinationthereof (e.g., human identifies vertices of polylines, machine generatespolygons using polygon rasterizer). In some examples, for each inputimage 302, there may be a corresponding label 308.

As illustrated in FIG. 3A, the machine learning model(s) 108 may betrained using both down-sampled (e.g., after down-sampling 304) versionsand region of interest (ROI) versions (e.g., after cropping 306) of theinput images 302. In some examples, the input image(s) 302 may undergodown-sampling 304. For example, down-sampling 304 may includedown-sampling the resolution of the input image 302 by some amount, suchas a quarter, a third, a half, a tenth, etc. The ROI images mayrepresent a cropped image (e.g., center crop, half size, etc.) generatedafter the input images 302 undergo cropping 306. The input images 302may be cropped to include an area around the vanishing point, ahorizontal stripe at a perspective view corresponding to a certaindistance at a bird's eye view, a center crop of a higher resolution toprovide more information at farther distance, another cropped region,and/or a combination thereof. In an example, the images generated bydown-sampling 304 and cropping 306 may be of different resolutionsand/or different portions of the fields of view than the input images302. In such an example, the machine learning model(s) 108 may learnfeatures from a variety of image resolutions and portions of fields ofview. The images generated by down-sampling 304 and cropping 306 maythen be grouped together in one or more data batches or datamini-batches 314 as training images to be input to the machine learningmodel(s) 108. In some examples, the size of the data mini-batch(es) 314may be a tunable hyper-parameter. In one example, the datamini-batch(es) 314 may include an equal number of images fromdown-sampling 304 and cropping 306.

Referring to FIG. 3B, FIG. 3B includes an example of down-sampling 304and cropping 306 the input images 302. As illustrated in FIG. 3A, afull-resolution image 322 (e.g., one of the input images 302) may bedown-sampled to create a first lower-resolution image 324A. The firstlower-resolution image 326A may then be down-sampled again, in someexamples, to create the second lower-resolution image 324B from thefirst lower-resolution image 324A. Although two stages of down-sampling304 are illustrated in FIG. 3B, this is not intended to be limiting, andany number of stages may be used. After down-sampling 304, the resultingimages may be used as a first half of the mini-batch 328A for trainingimages 328.

The full-resolution image 322 may also undergo cropping 306 to create acropped image 324B. The portion of the image that is cropped maycorrespond to cropped portion 352 included for illustrative purposesonly in the full-resolution image 322. The cropped image 324B may, insome examples, be down-sampled (e.g., as part of down-sampling 304) togenerate a lower-resolution cropped image 326B. Although only a singledown-sampling is illustrated, this is not intended to be limiting, andany number of down-sampling stages may be included on the cropped image324B. After cropping 306, the resulting images may be used as a secondhalf of the mini-batch 328B for training images 328.

In some examples, the images from down-sampling (e.g., 326B) and theimages from cropping (e.g., 326B) may be down-sampled to the sameresolution for use as the training images 328. In some examples, thelower-resolution cropped image 326B may be padded (e.g., may havezero-valued pixels added) to create an image of the same resolution asthe second lower-resolution image 326A. In other examples, the imagesmay be of different resolutions. In addition, in some examples, cropping306 and down-sampling 304 may be performed simultaneously, or may beperformed at different times.

The labels 308 corresponding to the input images 302 may undergo similardown-sampling 310 and cropping 312 as their corresponding input images302. Down-sampling 310 may be applied to the labels 308 to reduce theinput resolution of the labels 308. In some examples, the down-sampling310 may reduce the resolution of the labels 308 to the same resolutionas down-sampling 304 of the corresponding input images 302. Cropping 312may be applied to the labels 308 of the input images 302 that werecropped to generate cropped labels. In some examples, cropping 312 maycrop the labels 308 in a same manner as cropping 306 of thecorresponding input images 302. In other examples, cropping 312 may notbe a direct, one-to-one correlation to cropping 306, such as describedherein with respect to FIGS. 4A-4C. The labels generated usingdown-sampling 310 and cropping 312 may be combined in a datamini-batch(es) 314 similar to data mini-batch(es) of their correspondinginput images 302, and may be used as ground truth data (e.g., afteronline data augmentation 316, in examples) for training the machinelearning model(s) 108 (e.g., for comparison to the segmentation mask(s)110 output by the machine learning model(s) 108 using a loss function318).

In some non-limiting examples, the mini-batch(es) 314, including theimages and corresponding labels, may undergo online data augmentation316 to transform the images and corresponding labels in themini-batch(es) 314. In examples, the training images 328 may undergo oneor more spatial transformations (e.g., left flip, right flip, zoom in, azoom out, a random translation, etc.) and/or one or more colortransformations (e.g., hue, saturation, contrast, etc.), and thetraining labels 308 may undergo corresponding transformations.

For example, an input image 302 may be associated with a label 308including vertices of a polygon (e.g., an annotation of vertices forlabels 308) representing a lane and/or a boundary. The input image 302(or the down-sampled or cropped version thereof) may undergo atransformation(s) to generate a transformed image. The vertices of thepolygon (e.g., labels 308, or the down-sampled or cropped versionthereof) may similarly undergo a transformation (e.g., change inlocation) to generate transformed vertices based on the transformationof the input image 302. For example, if the input image 302 is rotatedthirty degrees, the vertices of the polygon may be transformed tocorrespond to a location of the vertices in the input image 302 asrotated thirty degrees.

For any input image 302 and/or transformed image, a masked image maythen be generated by masking the portion of the transformed imagecorresponding to the transformed label (e.g., a transformed polygon maybe masked that corresponds to the transformed vertices). The machinelearning model(s) 108 may be trained using the masked image(s) as groundtruth data. A result of online data augmentation 316 is reducing thelikelihood of overfitting of the trained machine learning model(s) 108to the training images and labels, thereby generating a more usefulmodel(s) 108 for deployment in a real-world scenario.

The augmented images generated by online data augmentation 316 may bepassed through the machine learning model(s) 108. In some examples, themachine learning model(s) 108 (e.g., a neural network(s)) may be trainedwith original and/or augmented images using multiple iterations untilthe value of loss function(s) 318 of the machine learning model(s) 108is below a threshold loss value. The machine learning model(s) 108 maybe trained to generate segmentation mask(s) 110 for each of the originaland augmented images. The segmentation mask(s) 110 output by the machinelearning model(s) 108 may represent portions of the original and/oraugmented image(s) determined to correspond to lane markings, roadboundaries, crosswalks, intersections, and/or other features of adriving surface of the vehicle 800.

In some examples, the segmentation mask(s) 110 may include points (e.g.,pixels) of the image(s) where lanes, boundaries, and/or other featuresare determined to be located by the machine learning model(s) 108. Insome examples, the segmentation mask(s) 110 generated may be a binarymask(s) (e.g., the binary mask head 334 of FIG. 3C) with a firstrepresentation for background elements (e.g., elements other than lanesand boundaries) and a second representation for foreground elements(e.g., lanes and boundaries). In other examples, the machine learningmodel(s) 108 may be trained to generate a multi-class segmentationmask(s) (e.g., the multi-class mask head 332 of FIG. 3C) as thesegmentation mask(s) 110, with different classes relating to differentlane markings, boundaries, and/or other features. In such examples, theclasses may include a first class for background elements, a secondclass for road boundaries, a third class for solid lane markings, afourth class for dashed lane markings, a fifth class for intersections,a sixth class for crosswalks, a seventh class for lane splits, and/orother classes for other features.

The machine learning model(s) 108 may perform forward pass computationson the original and/or augmented images. In some examples, the machinelearning model(s) 108 may extract features of interest from the image(s)and predict a probability of a boundary class, a lane marking class, oranother feature class in the images (e.g., on a pixel-by-pixel basis).The loss function 318 may be used to measure loss (e.g., error) in thesegmentation mask(s) 110 (e.g., predictions generated by the machinelearning model(s) 108) as compared to the ground truth data (e.g., theoriginal and/or augmented labels, annotations, and/or masks). In oneexample, a binary cross entropy function may be used as the lossfunction 318. In any example, backward pass computations may beperformed to recursively compute gradients of the loss function withrespect to training parameters. In some examples, weight and biases ofthe machine learning model(s) 108 may be used to compute thesegradients. For example, region based weighted loss may be added to theloss function 318, where the loss function 318 may increasingly penalizeloss at farther distances from the bottom of the image, as describedherein. In such a case, the region-based weight loss may be representedas follows in equation (1):

$\begin{matrix}{{{Weighted}{Loss}} = {{- \frac{1}{H*W}}{\sum}_{h = 0}^{H - 1}{\sum}_{w = 0}^{W - 1}{{Weight}_{h,w}\left\lbrack {{y_{{t{rue}},h,w}*\log\left( y_{{{pre}d},h,w} \right)} + {\left( {1 - y_{{true},h,w}} \right)*\log\left( {1 - y_{{pred},h,w}} \right)}} \right\rbrack}}} & (1)\end{matrix}$

where Weight_(h,w) represents the weight coefficients alongtwo-dimensional, height (h)×width (w), output segmentation mask(s) 110,Y_(true) represents the ground truth location, and Y_(pred) representsthe predicted location output by the machine learning model(s) 108. Insome examples, Y_(true) may be 0 or 1 and Y_(pred) may be a float valuebetween 0 and 1. A vanilla loss function, a special case whereWeight_(h,w)=1 for all h, w, treats every region the same. However, aregion based weighted loss function penalizes more errors on the fardistance, and therefore improves the detection in the far distance. Insome examples, the weighted coefficients may be set as a function of rownumbers of the images.

In some examples, the machine learning model(s) 108 may be trained usingboth a multi-class mask head 332 (e.g., multi-class mask includingheads, 332A-332C, for each class) and a binary mask head 334. In someexamples, the binary mask head 334 may be derived from the multi-classmask head 332. However, deriving the binary mask head 334 from themulti-class mask head 332 may result in a binary mask head 334 that doesnot perform as well as a separately trained binary mask head 334.

As such, with reference to FIG. 3C, both the multi-class mask head 332and the binary mask head 334 may be trained (e.g., jointly). Theoriginal and/or augmented images may be input to the machine learningmodel(s) 108, and the machine learning model(s) 108 may be trained tooutput both the multi-class mask head 332 and the binary mask head 334.In some non-limiting examples, the multi-class mask head loss mayinclude independent binary cross entropy for each class in themulti-class mask. In that case, the loss function 318 may be calculatedas a total loss (equation (4)) of the multi-class mask head loss(equation (2)) and the binary mask head loss (equation (3)), as follows:

${{Multiclass}{Loss}} = {{- \frac{1}{C*H*W}}{\sum\limits_{c = 0}^{C - 1}{\sum\limits_{h = 0}^{H - 1}{\sum\limits_{w = 0}^{W - 1}\left\lbrack {{y_{{t{rue}},c,h,w}*\log\left( y_{{pred},c,h,w} \right)} + {\left( {1 - y_{{true},c,h,w}} \right)*\log\left( {1 - y_{{pred},c,h,w}} \right)}} \right\rbrack}}}}$$\begin{matrix}{{{Multiclass}{Loss}} = {{- \frac{1}{C*H*W}}{\sum}_{c = 0}^{C - 1}{\sum}_{h = 0}^{H - 1}{{\sum}_{w = 0}^{W - 1}\left\lbrack {{y_{{t{rue}},c,h,w}*\log\left( y_{{pred},c,h,w} \right)} + {\left( {1 - y_{{true},c,h,w}} \right)*\log\left( {1 - y_{{pred},c,h,w}} \right)}} \right\rbrack}}} & (2)\end{matrix}$ $\begin{matrix}{{{Binary}{Loss}} = {{- \frac{1}{H*W}}{\sum}_{h = 0}^{H - 1}{{\sum}_{w = 0}^{W - 1}\left\lbrack {{y_{{true},h,w}*\log\left( y_{{pred},h,w} \right)} + {\left( {1 - y_{{true},h,w}} \right)*\log\left( {1 - y_{{predh},w}} \right)}} \right\rbrack}}} & (3)\end{matrix}$ $\begin{matrix}{{{Total}{Loss}} = {{a*{multiclass\_ head}{\_ loss}} + {{binary}{}{Mask\_ head}{\_ loss}}}} & (4)\end{matrix}$

where C is the number of classes of the multi-class mask head (e.g.,multi-class mask), H is the height of the output tensor, W is the widthof the output tensor, and a is the loss weight between the two heads. Insome examples, a may be a hyper-parameter that may be optimized.

In another non-limiting example, multi-class mask head loss may usemulti-class cross entropy (equation (5)) or a weighted cross entropy(equation (6)) calculated as follows:

$\begin{matrix}{{{Multiclass}{Loss}} = {{- \frac{1}{C*H*W}}{\sum}_{c = 0}^{C - 1}{\sum}_{h = 0}^{H - 1}{{\sum}_{w = 0}^{W - 1}\left\lbrack {y_{{t{rue}},c,h,w}*\log\left( t_{{pred},c,h,w} \right)} \right\rbrack}}} & (5)\end{matrix}$ $\begin{matrix}{{{Weighted}{Loss}} = {{- \frac{1}{C*H*W}}{\sum}_{c = 0}^{C - 1}{Weight}_{c}{\sum}_{h = 0}^{H - 1}{{\sum}_{w = 0}^{W - 1}\left\lbrack {y_{{t{rue}},c,h,w}*\log\left( y_{{pred},c,h,w} \right)} \right\rbrack}}} & (6)\end{matrix}$

Now referring to FIG. 3D, each block of method 340, described herein,may comprise a computing process that may be performed using anycombination of hardware, firmware, and/or software. For instance,various functions may be carried out by a processor executinginstructions stored in memory. The methods may also be embodied ascomputer-usable instructions stored on computer storage media. Themethods may be provided by a standalone application, a service or hostedservice (standalone or in combination with another hosted service), or aplug-in to another product, to name a few. In addition, method 340 isdescribed, by way of example, with respect to the vehicle 800 and theprocess 300. However, these methods may additionally or alternatively beexecuted by any one system, or any combination of systems, including,but not limited to, those described herein.

FIG. 3D is a flow diagram showing a method 340 for training a neuralnetwork to detect lanes and boundaries using transformed images andlabels, in accordance with some embodiments of the present disclosure.The method 340, at block B342, includes receiving image datarepresentative of an image of a driving surface. For example, image datamay be received that is representative of input images 302, where theinput images 302 include representations of driving surfaces of thevehicle 800. In some examples, the input images 302 may be stored in adata store or a repository.

The method 340, at block B344 includes receiving annotationscorresponding to locations of at least one of lane markings orboundaries of the driving surface. For example, labels 308 correspondingto the input images 302 may be received. The labels 308 may includeground truth information corresponding to lanes, boundaries, and/orother features of the driving surface represented in the correspondinginput images 302.

The method 340, at block B346 includes applying one or moretransformations to the image to generate a transformed image. Forexample, online data augmentation 316 (e.g., spatial transformations,color transformations, etc.) may be performed on the input images 302 togenerate transformed images.

The method 340, at block B348 includes applying one or more secondtransformations corresponding to the one or more first transformationsto each of the annotations to generate transformed annotations. Forexample, online data augmentation 316 may be performed on the labels 308to generate augmented labels. In some examples, the labels 308 may betransformed using similar or corresponding transformation as thetransformations of the input images 302.

The method 340, at block B350 includes training a neural network usingthe image, the annotations, the transformed image, and the transformedannotations. For example, the machine learning model(s) 108 may betrained to identify pixels within training images 328 (e.g.,down-sampled and/or cropped versions of the input images 302), augmentedtraining images, training labels (e.g., down-sampled and/or croppedversions of the labels 308), and/or augmented training labels. Thetraining labels and augmented training labels may be used as groundtruth data to train the machine learning model(s) 108 to detectcorresponding lane marking, road boundaries, and/or other features ofthe driving surface.

Referring to FIG. 4A, FIG. 4A is an illustration of an example process400 for generating ground truth data to train a machine learningmodel(s) to detect lanes, road boundaries, and/or other features, inaccordance with some embodiments of the present disclosure. For example,online data augmentation 316 may be performed on polygon verticesreceived as labels 308 during ground truth generation. When an inputimage 302 associated with the labels 308 goes through down-sampling 304,the same down-sampling may be applied to the labels 308 (e.g., thepolygon vertices) corresponding to the input image 302. Similarly, whenan input image 302 goes associated with the labels 308 goes throughcropping 306, the labels 308 must be adjusted. The polygons (e.g.,polygon 470) generated from the polygon vertices, or other label types(e.g., lines) from the labels 308, may represent lanes, road boundaries,and/or other features depicted in the corresponding input images 302.

Full-resolution images 410A and 410B may include the polygon 470 thatmay be in full-resolution. The polygon 470, as described herein, may begenerated from polygon vertices as part of the ground truth maskgeneration, or may be generated in a single step (e.g., drawn orotherwise generated without first having vertices). Generating polygonsfrom polygon vertices to represent ground truth data for lane markings,road boundaries, and/or other features is further described herein, atleast with respect to FIGS. 5A and 5B. Although FIGS. 4A-4D aredescribed with respect to polygons as the ground truth mask, this is notintended to be limiting. For example, labels 308 of other shapes (e.g.,lines, circles, amorphous shapes, etc.) may be used without departingfrom the scope of the present disclosure.

The full-resolution image 410A and the associated polygon 470 (e.g., aground truth mask) may be down-sampled (e.g., using down-sampling 304)to generate a down-sampled image 412 with a down-sampled version of thepolygon 472. For example, the full-resolution image 410A may bedown-sampled, and then the down-sampling of the full-resolution image410A may be used to inform the down-sampling of the polygon vertices,and the down-sampled polygon 472 may be generated from the down-sampledpolygon vertices. The down-sampled polygon 472 may then be masked torepresent the portion of the down-sampled image 412 that corresponds tothe lane marking, boundary, and/or other feature. In some examples, themask may be generated by using a canvas to mask out pixels (or portionsof the down-sampled image 412) outside of the down-sampled polygon 472.The mask may correspond to a binary mask (e.g., as illustrated in maskedimage 414), where the down-sampled polygon 472 may be a first color(e.g., white) and the pixels outside of the down-sampled polygon are asecond color (e.g., black). However, the masked image 414 may also beused to train a multi-class mask, as described herein. The masked image414 may then be used as ground truth data to train the machine learningmodel(s) 108 to detect lane markings, road boundaries, and/or featuresthat correspond to the location of the down-sampled polygon 472 inreal-world coordinates.

The full-resolution image 410B and the associated polygon 470 (e.g., aground truth mask) may be cropped (e.g., using cropping 306) to generatea cropped or ROI image 416. However, after cropping 306, thecorresponding labels 308 (e.g., the polygon 470) must also be adjustedor transformed to map to the ROI image 416. For example, while cropping306 the polygon 470, it may be determined that a portion of the polygon470 is outside of the ROI image 416 (e.g., extends beyond ROI 474 of thefull-resolution image 410B used to generate the cropped image 416). Insuch examples, a canvas may be used to augment the polygon vertices(e.g., four corners of the polygon 470) such that the canvas masks outthe pixels of the polygon outside of the ROI 474, as can be seen by theportion of the polygon 470 in masked image 418. Once the updated (e.g.,within the ROI 474) polygon vertices are determined, any augmentationsor transformations to the cropped image 416 may also be applied to theupdated polygon vertices, as described herein. The masked image 418 maythen be used as ground truth data to train the machine learning model(s)108 to detect lane markings, road boundaries, and/or other features.

Now referring to FIG. 4B, FIG. 4B is an illustration of an exampleprocess 420 for performing online data augmentation and cropping ofground truth masks, in accordance with some embodiments of the presentdisclosure. The full-resolution image 410B with the polygon 470 (e.g.,as a label 308) may be received that corresponds to an input image 302.The full-resolution image 410B may include the ROI 474 for cropping 306to generate a cropped image. One or more spatial transformations (e.g.,zoom-in, zoom-out, flip right, flip left, etc.) may be performed on thecropped image and correspondingly to the polygon 470 (e.g., to thevertices of the polygon using online data augmentation 316) to generatea transformed image 422 with a transformed polygon 476. The spatialtransformation illustrated in FIG. 4B may be a zoom-out transformation,however, the spatial transformations are not so limited. An augmented ortransformed ground truth mask 424 may be generated within the ROI 474 bydetermining the transformed polygon vertices and masking out the regionoutside of the ROI 474.

In some examples, the augmented ground truth mask 424 may be padded(e.g., may have zero-valued pixels added to increase the spatialresolution) to a desired size for the training images 328. In someexamples, the padding is implemented using a canvas, such that theaugmented ground truth mask sits within the canvas, and the canvasprovides the padding. Ultimately, a padded image 426 may be used asground truth data for training the machine learning model(s) 108 todetect lane markings, road boundaries, and/or other features.

Now referring to FIGS. 4C-4D, each block of methods 440 and 460,described herein, may comprise a computing process that may be performedusing any combination of hardware, firmware, and/or software. Forinstance, various functions may be carried out by a processor executinginstructions stored in memory. The methods may also be embodied ascomputer-usable instructions stored on computer storage media. Themethods may be provided by a standalone application, a service or hostedservice (standalone or in combination with another hosted service), or aplug-in to another product, to name a few. In addition, methods 440 and460 are described, by way of example, with respect to the vehicle 800and the processes 300, 320, 400 and/or 420. However, these methods mayadditionally or alternatively be executed by any one system, or anycombination of systems, including, but not limited to, those describedherein.

FIG. 4C is a flow diagram showing a method 440 for training a neuralnetwork to detect lanes and boundaries using down-sampled images, inaccordance with some embodiments of the present disclosure. The method440, at block B442, includes receiving a first image in a first imageresolution and associated first vertex data corresponding to firstlocations of vertices of a polygon associated with the first image. Forexample, an input image 302 (e.g., the full-resolution image 410A) andassociated labels 308 may be received, where the input image 302 has afirst resolution and the labels 308 are represented as first vertex datacorresponding to a polygon 470 of a lane, boundary, or other feature.

The method 440, at block B444, includes applying a first transform tothe first image to generate a second image in a second image resolution.For example, down-sampling 304 may be applied to the input image 302 toconvert the input image 302 to the first lower resolution image 324Aand/or the second lower resolution image 326A.

The method 440, at block B446, includes applying a second transform tothe first vertex data to generate second vertex data corresponding tosecond locations of vertices of a second polygon within the second imagebased at least in part on the first transform. For example, a second setof vertex data for a down-sampled polygon 472 may be generated bydown-sampling the polygon 470 corresponding to the down-sampling of theinput image 302.

The method 440, at block B448, includes generating the second polygonfor the second image based at least in part on the second vertex data.For example, the down-sampled polygon 472 may be generated based on thesecond vertex data.

The method 440, at block B450, includes masking at least a portion ofthe second image that corresponds to the second polygon to generate amasked image. For example, a binary mask in lower resolution 414 may begenerated based on the second set of vertices by masking thedown-sampled polygon 472 to generate the masked image 414.

The method 440, at block B452, includes training a neural network usingground truth data that corresponds to the masked image. For example, themachine learning model(s) 108 may be trained using the masked image 414as ground truth data.

Now referring to FIG. 4D, FIG. 4D is a flow diagram showing a method 460for training a neural network to detect lanes and boundaries usingcropped images, in accordance with some embodiments of the presentdisclosure. The method 460, at block B462, includes receiving an imageand associated vertex data corresponding to locations of vertices of apolygon within the image. For example, an input image 302 (e.g., thefull-resolution image 410B) and associated labels 308 may be receivedwhere the labels 308 are represented as vertex data corresponding tolocations of vertices of a polygon 470 within the input image 302.

The method 460, at block B464, includes determining a crop portion ofthe image, at least a portion of the polygon being outside of the cropportion. For example, the ROI 474 may be determined, and the croppedimage 416 may be generated. After cropping 306, at least a portion ofthe polygon 470 may be outside of the ROI 474.

The method 460, at block B466, includes masking at least a portion ofthe image that corresponds to the polygon and the crop portion of theimage to generate a masked image. For example, a first mask may begenerated to mask out the portion of the polygon 470 that is outside ofthe ROI 474 and a second mask may be generated within the ROI 474 tomask the portion of the cropped image 416 that corresponds to the lane,boundary, and/or other feature. The first and second masks may beapplied to generate the masked image 418.

The method 460, at block B468, includes training a neural network usingground truth data that corresponds to the masked image. For example, themachine learning model(s) 108 may be trained using the masked image 418as ground truth data.

Annotating Features of a Driving Surface

In some examples, annotations of features of a driving surface, such aslanes, boundaries, crosswalks, intersections, lane merges, lane splits,and/or other features may be used to generate ground truth data fortraining a machine learning model(s). For example, an image may bereceived along with annotations indicating marked vertices representingvertices of a polyline. Line segments (e.g., polylines) may then begenerated to join adjacent vertices. The polylines may then be expandedinto corresponding polygons. As a non-limiting example, for each of thevertices, an adjacent vertex may be generated such that the adjacentvertex is perpendicular to a polyline extending from the vertex. A setof second polylines may then be generated between the vertex and theadjacent vertex, and a set of third polylines may then be generatedbetween corresponding adjacent vertices. The first polyline, the secondpolylines, and the third polyline may form a polygon that may representa portion of a boundary, lane, or other feature of a driving surface,and together the polygons may represent the boundary, lane, and/or otherfeature of the driving surface (e.g., as a ground truth mask). In someexamples, the width of the polygons (e.g., the length of the set ofsecond polylines) may be based on the distance of the correspondingsecond polyline from the bottom of the image (e.g., relative to thecamera or other sensor). In such examples, the width of the polygons,and the corresponding length of the second polylines, may decrease fromthe bottom of the image toward the top.

By generating the polygons in this way for ground truth data, theaccuracy of the machine learning model(s) may be improved, and theefficiency of creating the ground truth data may be increased. Forexample, because polylines are generated from vertices, the resultingpolyline is already more accurate than a fully hand drawn or computergenerated line, and by expanding the polylines to polygons, the accuracyis further increased.

In some embodiments, these polygons may be generated or rendered withrespect to an image to generate a visible, rendered line that delineatesa road boundary, a lane marking, and/or another feature, even when noroad boundary or lane markings are actually present in the image (e.g.,due to occlusion from other objects, from weather, when non-existent,etc.). For example, by generating ground truth data in this way, themachine learning model(s) may be trained to recognize transitions frompavement to a different surface (e.g., dirt, gravel, sand, etc.),pavement to a concrete barrier, pavement to a curb, gravel to dirt, etc.Advantageously, this technique may also allow for training the machinelearning model(s) to generate full and accurate detection of lanes andboundaries even for adverse, or non-ideal, weather and road conditions.

Now referring to FIG. 5A, FIG. 5A is an illustration of an exampleprocess 500 for annotating road boundaries for generating ground truthdata, in accordance with some embodiments of the present disclosure. Theannotation described with respect to FIG. 5A may relate to a roadboundary annotation, however, this is not intended to be limiting. Forexample, similar annotations may be made for lanes, crosswalks,intersections, parking lines, and/or other features of a drivingsurface, or other features of non-driving surfaces, such as fields,poles or other vertical or elongate structures, etc.

Road boundary annotation may include labeling a set of first vertices502A-502D, where the first vertices 502 may correspond to points (e.g.,pixels) within an image. In some examples, the vertices 502 may belabeled along a transition from one surface to another, as describedherein. In any example, the first vertices 502 may correspond to a roadboundary, a lane marking, and/or another feature of an environment(e.g., real-world or virtual) represented within an image. Firstpolylines may be generated between adjacent first vertices 502. Forexample, first polylines 516A-516C may be extended between each of theadjacent vertices 502A and 502B, 502B and 502C, 502C and 502D, and soon.

The first polylines 516 may then be expanded into polygons, such as520A, 520B and 520C. This may be done by generating second vertices504A-504D adjacent the first vertices 502 that correspond to the firstpolylines 516, the second vertices 504 may be spaced from the firstvertices 502 along a direction perpendicular to the first polylines 516(e.g., as indicated by the right angle, such as right angle 518). In onenon-limiting example, locations of the second vertices 504 with respectto the first vertices 502 may be determined based at least in part on adistance of the first vertices 502 from the bottom of the image. Forexample, the locations of the second vertices 504 may be determined suchthat that the second vertices 504 are spaced closer to the firstvertices as the distance of the first vertices 502 increases withrespect to the bottom of the image. For example, a location of thesecond vertex 504A may be a greater distance (e.g., along a lineperpendicular to the first polyline 516A) from the first vertex 502Athan the second vertex 504B is placed from the first vertex 502B (e.g.,along a line perpendicular to the first polyline 516B), and so on. Insome examples, the locations of the second vertices 504 may bedetermined based on a slope of the previous polyline or first polyline516. For example, the location of the second vertex 504B may bedetermined based on the slope of the first polyline 516A joining firstvertices 502A and 502B. By generating polygons 520 such that thepolygons 520 are gradually less wide as the distance from the bottom ofthe image increases, the accuracy of the machine learning model(s) 108,especially at predicting lanes, boundaries, and/or other features at adistance, may be increased.

Second polylines 522 may be generated that extend between adjacentsecond vertices 504. For example, a line segment may be extended betweeneach of the adjacent vertices 504A and 504B, 504B and 504C, 504C and504D, and so on, to generate the second polylines 522.

Third polylines 524 may be generated that extend between correspondingfirst vertices 502 and second vertices 504. For example, line segmentsmay be extended between vertices 502A and 504A, 502B and 504B, 502C and504C, and 502D and 504D, and so on to generate the third polylines 524.The polygons 520 may then be used as annotations for labels 308 ofground truth data to train the machine learning model(s) 108 to detectboundaries, roads, and/or other features.

In some examples, the first vertices 502 and the second vertices 504 maybe determined, and the vertices may be used as the vertex data—asdescribed herein—that may be augmented during augmentation ortransformation of the images and labels. As such, the vertices 502 and504 may be used as preliminary annotations, and after augmentation ofthe vertices 502 and 504, the polygons 520 may then be generated. Inother examples, the polygons 520 may be generated, and the entirepolygon may undergo the augmentations and/or transformations.

Now referring to FIG. 5B, FIG. 5B is a diagram illustrating an exampleroad boundary annotation 510, in accordance with some embodiments of thepresent disclosure. A road boundary may be any visible boundary at theedge of the road that is drivable. Road boundaries may also includetransitions from pavement to dirt, pavement to a concrete barrier orcurb, etc. While training the machine learning model(s) 108, a polygonmay be drawn to include a visible line that indicates road boundaries asdescribed herein, at least with respect to FIG. 5A. In some examples,the machine learning model(s) 108 may be trained using annotations forlanes and boundaries on a road without lane or boundary markings In suchan example, road boundaries may be annotated as labels 308 (e.g., lines512 and 514) on roads where no markings are visible. This may enable themachine learning model(s) 108 to learn a variety of patterns for roadboundaries, lane markings, and/or other features.

In some examples, road boundaries may be annotated as polygons using theprocess described with respect to FIG. 5A, even when there areconflicting lane markings in the images. For example, boundaries may beannotated when there is a solid yellow or solid white line at the end ofthe road, or when other lines are very close in the images. This mayenable the machine learning model(s) 108 to learn different patterns andto enable lane-keeping on a road with incorrect lane markings, which maybe more common on local and/or rural roads (e.g., surface streets).

A variety of annotations may be used during training to train themachine learning model(s) 108 to accurately detect occluded lanemarkings and boundaries, in addition to those included herein. Forexample, the machine learning model(s) 108 may be trained usingannotations for extending lanes beyond or through a vehicle or otherobject on the driving surface that may be occluding at least a portionof the lane marking, boundary, and/or other feature. The machinelearning model(s) 108 may also be trained using images and labelsannotating lanes and boundaries occluded as result of weather or roadconditions, such as during rain and/or when snow is covering the drivingsurface. As such, the current system may be able to accurately detectlanes and boundaries in real-time, even with less than ideal weatherand/or road conditions.

Now referring to FIG. 5C, each block of method 540, described herein,comprises a computing process that may be performed using anycombination of hardware, firmware, and/or software. For instance,various functions may be carried out by a processor executinginstructions stored in memory. The methods may also be embodied ascomputer-usable instructions stored on computer storage media. Themethods may be provided by a standalone application, a service or hostedservice (standalone or in combination with another hosted service), or aplug-in to another product, to name a few. In addition, method 540 isdescribed, by way of example, with respect to the vehicle 800 and theprocess 500. However, these methods may additionally or alternatively beexecuted by any one system, or any combination of systems, including,but not limited to, those described herein.

FIG. 5C is a flow diagram showing a method 540 for annotating roadboundaries for ground truth generation, in accordance with someembodiments of the present disclosure. The method 540, at block B542,includes receiving image data representative of an image of anenvironment. For example, image data may be received that representedone or more images of a physical environment including a drivingsurface(s).

The method 540, at block B544, includes generating a polygon. Forexample, the polygons 520 may be generated using block B544A-B544E,described herein.

The method 540, at block B544A, includes identifying first vertices thatcorrespond to a boundary within the environment. For example, the firstvertices 502 may be identified. The identification may be as a result ofgeneration of the first vertices 502 using one or more programs, aresult of data representing locations of the first vertices 502 beingreceived, and/or another method.

The method 540, at block B544B, includes generating polylines betweenadjacent first vertices. For example, the first polylines 516 may begenerated by extending line segments between each of adjacent vertices502A and 502B, 502B and 503C, and 502C and 502D, and so on.

The method 540, at block B544C, includes generating second verticesadjacent the first vertices spaced along a direction perpendicular tothe first polyline. For example, the second vertices 504 may begenerated adjacent the first vertices, and may be spaced from the firstvertices 502 along a direction perpendicular to the first polylines 516.

The method 540, at block B544D, includes generating a second polylineextending between the second vertices. For example, second polylines 522may be generated that extend between each of the adjacent secondvertices 504.

The method 540, at block B544E, includes generating third polylinesextending between corresponding first vertices and second vertices. Forexample, the third polylines 524 may be generated to extend betweenadjacent first vertices 502 and second vertices 504.

Referring to FIG. 6A, FIG. 6A is a diagram illustrating an examplecrosswalk and intersection annotation, in accordance with someembodiments of the present disclosure. The machine learning model(s) 108may be trained to detect combined crosswalks and intersections byannotating the regions corresponding to the crosswalk and theintersection separately. For example, the crosswalk may be annotatedwith a “crosswalk” label/class using the solid lines 602A-602E (e.g.,which may be polygons generated similarly to the process of FIGS. 5A-5C)surrounding the crosswalk markings in the image. On the other hand,intersection may be annotated with an “intersection” label/class usingdotted or broken lines 604A-604D (e.g., which may be polygons generatedsimilarly to the process of FIGS. 5A-5C) separate from the crosswalkannotation lines 602A-602E. In some examples, each intersection line maybe labeled individually. For example, 604A and 604B may be labeled asone intersection class, and 604C and 604D may be labeled as anotherintersection class. In another example, a crossing intersection (e.g.,intersection at cross traffic) may be labeled as a separate “crossintersection” class than a through or opposite traffic intersection. Assuch, the machine learning model(s) 108 may be trained to learn todetect multiple lane and boundary markings even when more than oneexists in the same general area with similarly functioning labels. Eventhough FIG. 6A depicts crosswalk and intersection labeling, this is forexample purposes only and is not intended to be limiting.

Now referring to FIGS. 6B and 6C, FIGS. 6B and 6C are diagramsillustrating example merge lane annotations, in accordance with someembodiments of the present disclosure. The machine learning model(s) 108may be trained to detect merge points by associating a merge point 622Cand/or 624B to a polygon annotation at the merge point. Merge points622C and 624B may denote the point where the merge begins (e.g., wherelane markings 622A and 622B meet with respect to FIG. 6B, and where lanemarking 624A ends with respect to FIG. 6C). A polygon may be generatedusing method 5C including the merge points. The merge point may beassociated with the corresponding polygons 622C and 624B. The machinelearning model(s) 108 may learn to detect merge points in real-time bybeing trained using such annotations.

Now referring to FIGS. 6D and 6E, FIGS. 6D and 6E are diagramsillustrating example split lane annotations, in accordance with someembodiments of the present disclosure. The machine learning model(s) 108may be trained to detect split points by associating a split point 626Cand/or 628C to a polygon annotation at the split point. Split points626C and 628C may denote the point where the split occurs (e.g., wherelane markings 626A and 626B merge with respect to FIG. 6D, and wherelane marking 628A ends with respect to FIG. 6E). A polygon may begenerated using method 5C including the split points. The split pointmay be associated with the corresponding polygons 626C and 628C. Themachine learning model(s) 108 may learn to detect split points inreal-time by being trained using such annotations.

Key Performance Indicators (KPI)

End-to-end performance of lane keeping may be evaluated by means of amean autonomous distance (MAD) metric. The MAD metric is the averagedistance an autonomous vehicle can drive without requiring humanintervention. The MAD metric can be measured by simulating driving onmultiple routes, and dividing the distance traveled without requiringhuman intervention by the number of system failures. In some examples,the MAD metric may be used as a KPI for the systems of the presentdisclosure. The current system may use KPI's for one or more differentintermediate components or modules to allow for isolating failures orproblems, quantifying focused improvements, and for quick systemiterations.

For example, referring to FIG. 7A, FIG. 7A is a diagram for an exampleperformance calculation at different regions within a field of view of asensor, as represented by an image 700. A basic performance metric for amachine learning model output is an fscore of the final output. FIG. 7Aillustrates a region fscore metric that may be performed by the currentsystem to determine fscores for different regions within an output ofthe machine learning model(s) 108 (e.g., different regions of the outputsegmentation mask(s) 110). The performance of the machine learningmodel(s) 108 may be calculated at any number of different regions,however the example of FIG. 7A includes three different regions. Thethree regions may include a close view region 702A, a mid view region70A, and a far view region 706. This allows the system to measureaccuracy with distance earlier in the pipeline than conventionalsystems. Given a camera lens spec of sensors of the vehicle 800, the rowindex of the segmentation mask(s) 110 in a perspective view can betranslated into the distance in world coordinates (e.g., 3-dimensionalreal-world coordinates). The segmentation mask(s) 110 may be dividedinto regions 702A, 704A and 706 in a variety of manners. For example,the segmentation mask(s) 110 may be divided into regions around thevanishing points, with respect to the center part of the road, withrespect to the sides of the road, ROI regions, etc. Two or more divisionmethods may be combined to divide the segmentation mask(s) 110 intoregions as well. An fscore of each region may be calculated separatelyfor accuracy.

Not referring to FIG. 7B, FIG. 7B is a diagram illustrating atwo-dimensional (2D) KPI measured from lane detection and ground truthpolyline points, in accordance with some embodiments of the presentdisclosure. The 2D KPI may be defined with respect to precision, recall,and average closest point distance measured between detection polylinepoints and ground truth polyline points for each label in labels 308.The detection points may be generated by resampling 112 of thesegmentation mask(s) 110. The ground truth polyline points may be pointsgenerated via online data augmentation 316. A distance thresholdfunction may be utilized to determine the closest point distance. In oneexample, the distance threshold function may be an algorithm defined asa function of image row. For each region 702A, 704A, and 706, averageclosest point distance between corresponding pixels of polylines in theground truth (e.g., 722A, 722D) and the detection (e.g., 722B, 722C) maybe calculated. If the average closest point distance is less than adistance threshold (e.g., 724A, 724B, 724C, and 724D) calculated usingthe distance threshold function, the precision test may be determined tobe successful. However, if the average closest point distance is morethan the distance threshold, the precision test may be determined to beunsuccessful. This same procedure may be followed to compute recall,with a distance threshold based on recall distance on the ground truthside. The precision and recall tests may be performed in each of thethree regions of the image(s).

Referring to FIG. 7C, FIG. 7C is a data flow diagram illustrating athree-dimensional (3D) KPI measured from lane detection and ground truthpolyline vertices (or points) in accordance with some embodiments of thepresent disclosure. The 2D pixel locations in both the ground truth mask744 (e.g. 414, 418, 426, etc.) and the detection masks 732 may beconverted to 3D real-world coordinates (e.g., GPS coordinates, GNSScoordinates, etc.). A mean minimum distance metric may be used tomeasure KPIs in the three regions of the lane detection masks. The meandistance metric may indicate the relativity of matching pixel points (orvertices) in ground truth and prediction. At the time of converting the2D pixel locations to 3D real-world coordinates, the ground truthpolyline vertices and prediction polyline vertices may be connected bylines—such as polynomial lines or clothoid curves. For example, thecorresponding pair of ground truth vertices and prediction vertices maybe scanned from near to far along the distance in the masks via Lane KPICalculation 748. If the scanning line is short, meaning, thecorresponding points are close in distance, such as within a certainthreshold distance, the pair may be determined to be a successfuldetection. However, if a ground truth intersection with a scanning lineis found but the prediction intersection is not, the ground truth pointmay be determined to be a false detection. The Lane KPI Calculation 748may include algorithms to generate KPI outputs 750 with respect todistance. In some examples, one or more of MMD 750A, success rate 750B,and miss rate 750C may be determined with respect to distance, asillustrated in KPI outputs 750. Success rate 750B may be calculated as apercentage of successful lane detection points. Miss rate 750C may becalculated as a percentage of missed or failed lane detection points. Ina not limiting example, these KPI outputs 750 may be generated for eachof the three regions. This may allow for performance calculations in avariety of ways in order to determine early and accurate failures of theseparate components of the lane detection system.

Example Autonomous Vehicle

FIG. 8A is an illustration of an example autonomous vehicle 800, inaccordance with some embodiments of the present disclosure. Theautonomous vehicle 800 (alternatively referred to herein as the “vehicle800”) may include a passenger vehicle, such as a car, a truck, a bus,and/or another type of vehicle that accommodates one or more passengers.Autonomous vehicles are generally described in terms of automationlevels, defined by the National Highway Traffic Safety Administration(NHTSA), a division of the US Department of Transportation, and theSociety of Automotive Engineers (SAE) “Taxonomy and Definitions forTerms Related to Driving Automation Systems for On-Road Motor Vehicles”(Standard No. J3016-201806, published on Jun. 15, 2018, Standard No.J3016-201609, published on Sep. 30, 2016, and previous and futureversions of this standard). The vehicle 800 may be capable offunctionality in accordance with one or more of Level 3-Level 5 of theautonomous driving levels. For example, the vehicle 800 may be capableof conditional automation (Level 3), high automation (Level 4), and/orfull automation (Level 5), depending on the embodiment.

The vehicle 800 may include components such as a chassis, a vehiclebody, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and othercomponents of a vehicle. The vehicle 800 may include a propulsion system850, such as an internal combustion engine, hybrid electric power plant,an all-electric engine, and/or another propulsion system type. Thepropulsion system 850 may be connected to a drive train of the vehicle800, which may include a transmission, to enable the propulsion of thevehicle 800. The propulsion system 850 may be controlled in response toreceiving signals from the throttle/accelerator 852.

A steering system 854, which may include a steering wheel, may be usedto steer the vehicle 800 (e.g., along a desired path or route) when thepropulsion system 850 is operating (e.g., when the vehicle is inmotion). The steering system 854 may receive signals from a steeringactuator 856. The steering wheel may be optional for full automation(Level 5) functionality.

The brake sensor system 846 may be used to operate the vehicle brakes inresponse to receiving signals from the brake actuators 848 and/or brakesensors.

Controller(s) 836, which may include one or more system on chips (SoCs)804 (FIG. 8C) and/or GPU(s), may provide signals (e.g., representativeof commands) to one or more components and/or systems of the vehicle800. For example, the controller(s) may send signals to operate thevehicle brakes via one or more brake actuators 848, to operate thesteering system 854 via one or more steering actuators 856, to operatethe propulsion system 850 via one or more throttle/accelerators 852. Thecontroller(s) 836 may include one or more onboard (e.g., integrated)computing devices (e.g., supercomputers) that process sensor signals,and output operation commands (e.g., signals representing commands) toenable autonomous driving and/or to assist a human driver in driving thevehicle 800. The controller(s) 836 may include a first controller 836for autonomous driving functions, a second controller 836 for functionalsafety functions, a third controller 836 for artificial intelligencefunctionality (e.g., computer vision), a fourth controller 836 forinfotainment functionality, a fifth controller 836 for redundancy inemergency conditions, and/or other controllers. In some examples, asingle controller 836 may handle two or more of the abovefunctionalities, two or more controllers 836 may handle a singlefunctionality, and/or any combination thereof.

The controller(s) 836 may provide the signals for controlling one ormore components and/or systems of the vehicle 800 in response to sensordata received from one or more sensors (e.g., sensor inputs). The sensordata may be received from, for example and without limitation, globalnavigation satellite systems sensor(s) 858 (e.g., Global PositioningSystem sensor(s)), RADAR sensor(s) 860, ultrasonic sensor(s) 862, LIDARsensor(s) 864, inertial measurement unit (IMU) sensor(s) 866 (e.g.,accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s),etc.), microphone(s) 896, stereo camera(s) 868, wide-view camera(s) 870(e.g., fisheye cameras), infrared camera(s) 872, surround camera(s) 874(e.g., 360 degree cameras), long-range and/or mid-range camera(s) 898,speed sensor(s) 844 (e.g., for measuring the speed of the vehicle 800),vibration sensor(s) 842, steering sensor(s) 840, brake sensor(s) (e.g.,as part of the brake sensor system 846), and/or other sensor types.

One or more of the controller(s) 836 may receive inputs (e.g.,represented by input data) from an instrument cluster 832 of the vehicle800 and provide outputs (e.g., represented by output data, display data,etc.) via a human-machine interface (HMI) display 834, an audibleannunciator, a loudspeaker, and/or via other components of the vehicle800. The outputs may include information such as vehicle velocity,speed, time, map data (e.g., the HD map 822 of FIG. 8C), location data(e.g., the vehicle's 800 location, such as on a map), direction,location of other vehicles (e.g., an occupancy grid), information aboutobjects and status of objects as perceived by the controller(s) 836,etc. For example, the HMI display 834 may display information about thepresence of one or more objects (e.g., a street sign, caution sign,traffic light changing, etc.), and/or information about drivingmaneuvers the vehicle has made, is making, or will make (e.g., changinglanes now, taking exit 34B in two miles, etc.).

The vehicle 800 further includes a network interface 824, which may useone or more wireless antenna(s) 826 and/or modem(s) to communicate overone or more networks. For example, the network interface 824 may becapable of communication over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. Thewireless antenna(s) 826 may also enable communication between objects inthe environment (e.g., vehicles, mobile devices, etc.), using local areanetwork(s), such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc.,and/or low power wide-area network(s) (LPWANs), such as LoRaWAN, SigFox,etc.

FIG. 8B is an example of camera locations and fields of view for theexample autonomous vehicle 800 of FIG. 8A, in accordance with someembodiments of the present disclosure. The cameras and respective fieldsof view are one example embodiment and are not intended to be limiting.For example, additional and/or alternative cameras may be includedand/or the cameras may be located at different locations on the vehicle800.

The camera types for the cameras may include, but are not limited to,digital cameras that may be adapted for use with the components and/orsystems of the vehicle 800. The camera(s) may operate at automotivesafety integrity level (ASIL) B and/or at another ASIL. The camera typesmay be capable of any image capture rate, such as 60 frames per second(fps), 820 fps, 240 fps, etc., depending on the embodiment. The camerasmay be capable of using rolling shutters, global shutters, another typeof shutter, or a combination thereof. In some examples, the color filterarray may include a red clear (RCCC) color filter array, a red clearblue (RCCB) color filter array, a red blue green clear (RBGC) colorfilter array, a Foveon X3 color filter array, a Bayer sensors (RGGB)color filter array, a monochrome sensor color filter array, and/oranother type of color filter array. In some embodiments, clear pixelcameras, such as cameras with an RCCC, an RCCB, and/or an RBGC colorfilter array, may be used in an effort to increase light sensitivity.

In some examples, one or more of the camera(s) may be used to performadvanced driver assistance systems (ADAS) functions (e.g., as part of aredundant or fail-safe design). For example, a Multi-Function MonoCamera may be installed to provide functions including lane departurewarning, traffic sign assist and intelligent headlamp control. One ormore of the camera(s) (e.g., all of the cameras) may record and provideimage data (e.g., video) simultaneously.

One or more of the cameras may be mounted in a mounting assembly, suchas a custom designed (3-D printed) assembly, in order to cut out straylight and reflections from within the car (e.g., reflections from thedashboard reflected in the windshield mirrors) which may interfere withthe camera's image data capture abilities. With reference to wing-mirrormounting assemblies, the wing-mirror assemblies may be custom 3-Dprinted so that the camera mounting plate matches the shape of thewing-mirror. In some examples, the camera(s) may be integrated into thewing-mirror. For side-view cameras, the camera(s) may also be integratedwithin the four pillars at each corner of the cabin.

Cameras with a field of view that include portions of the environment infront of the vehicle 800 (e.g., front-facing cameras) may be used forsurround view, to help identify forward facing paths and obstacles, aswell aid in, with the help of one or more controllers 836 and/or controlSoCs, providing information critical to generating an occupancy gridand/or determining the preferred vehicle paths. Front-facing cameras maybe used to perform many of the same ADAS functions as LIDAR, includingemergency braking, pedestrian detection, and collision avoidance.Front-facing cameras may also be used for ADAS functions and systemsincluding Lane Departure Warnings (“LDW”), Autonomous Cruise Control(“ACC”), and/or other functions such as traffic sign recognition.

A variety of cameras may be used in a front-facing configuration,including, for example, a monocular camera platform that includes a CMOS(complementary metal oxide semiconductor) color imager. Another examplemay be a wide-view camera(s) 870 that may be used to perceive objectscoming into view from the periphery (e.g., pedestrians, crossing trafficor bicycles). Although only one wide-view camera is illustrated in FIG.8B, there may any number of wide-view cameras 870 on the vehicle 800. Inaddition, long-range camera(s) 898 (e.g., a long-view stereo camerapair) may be used for depth-based object detection, especially forobjects for which a neural network has not yet been trained. Thelong-range camera(s) 898 may also be used for object detection andclassification, as well as basic object tracking.

One or more stereo cameras 868 may also be included in a front-facingconfiguration. The stereo camera(s) 868 may include an integratedcontrol unit comprising a scalable processing unit, which may provide aprogrammable logic (FPGA) and a multi-core micro-processor with anintegrated CAN or Ethernet interface on a single chip. Such a unit maybe used to generate a 3-D map of the vehicle's environment, including adistance estimate for all the points in the image. An alternative stereocamera(s) 868 may include a compact stereo vision sensor(s) that mayinclude two camera lenses (one each on the left and right) and an imageprocessing chip that may measure the distance from the vehicle to thetarget object and use the generated information (e.g., metadata) toactivate the autonomous emergency braking and lane departure warningfunctions. Other types of stereo camera(s) 868 may be used in additionto, or alternatively from, those described herein.

Cameras with a field of view that include portions of the environment tothe side of the vehicle 800 (e.g., side-view cameras) may be used forsurround view, providing information used to create and update theoccupancy grid, as well as to generate side impact collision warnings.For example, surround camera(s) 874 (e.g., four surround cameras 874 asillustrated in FIG. 8B) may be positioned to on the vehicle 800. Thesurround camera(s) 874 may include wide-view camera(s) 870, fisheyecamera(s), 360 degree camera(s), and/or the like. Four example, fourfisheye cameras may be positioned on the vehicle's front, rear, andsides. In an alternative arrangement, the vehicle may use three surroundcamera(s) 874 (e.g., left, right, and rear), and may leverage one ormore other camera(s) (e.g., a forward-facing camera) as a fourthsurround view camera.

Cameras with a field of view that include portions of the environment tothe rear of the vehicle 800 (e.g., rear-view cameras) may be used forpark assistance, surround view, rear collision warnings, and creatingand updating the occupancy grid. A wide variety of cameras may be usedincluding, but not limited to, cameras that are also suitable as afront-facing camera(s) (e.g., long-range and/or mid-range camera(s) 898,stereo camera(s) 868), infrared camera(s) 872, etc.), as describedherein.

FIG. 8C is a block diagram of an example system architecture for theexample autonomous vehicle 800 of FIG. 8A, in accordance with someembodiments of the present disclosure. It should be understood that thisand other arrangements described herein are set forth only as examples.Other arrangements and elements (e.g., machines, interfaces, functions,orders, groupings of functions, etc.) may be used in addition to orinstead of those shown, and some elements may be omitted altogether.Further, many of the elements described herein are functional entitiesthat may be implemented as discrete or distributed components or inconjunction with other components, and in any suitable combination andlocation. Various functions described herein as being performed byentities may be carried out by hardware, firmware, and/or software. Forinstance, various functions may be carried out by a processor executinginstructions stored in memory.

Each of the components, features, and systems of the vehicle 800 in FIG.8C are illustrated as being connected via bus 802. The bus 802 mayinclude a Controller Area Network (CAN) data interface (alternativelyreferred to herein as a “CAN bus”). A CAN may be a network inside thevehicle 800 used to aid in control of various features and functionalityof the vehicle 800, such as actuation of brakes, acceleration, braking,steering, windshield wipers, etc. A CAN bus may be configured to havedozens or even hundreds of nodes, each with its own unique identifier(e.g., a CAN ID). The CAN bus may be read to find steering wheel angle,ground speed, engine revolutions per minute (RPMs), button positions,and/or other vehicle status indicators. The CAN bus may be ASIL Bcompliant.

Although the bus 802 is described herein as being a CAN bus, this is notintended to be limiting. For example, in addition to, or alternativelyfrom, the CAN bus, FlexRay and/or Ethernet may be used. Additionally,although a single line is used to represent the bus 802, this is notintended to be limiting. For example, there may be any number of busses802, which may include one or more CAN busses, one or more FlexRaybusses, one or more Ethernet busses, and/or one or more other types ofbusses using a different protocol. In some examples, two or more busses802 may be used to perform different functions, and/or may be used forredundancy. For example, a first bus 802 may be used for collisionavoidance functionality and a second bus 802 may be used for actuationcontrol. In any example, each bus 802 may communicate with any of thecomponents of the vehicle 800, and two or more busses 802 maycommunicate with the same components. In some examples, each SoC 804,each controller 836, and/or each computer within the vehicle may haveaccess to the same input data (e.g., inputs from sensors of the vehicle800), and may be connected to a common bus, such the CAN bus.

The vehicle 800 may include one or more controller(s) 836, such as thosedescribed herein with respect to FIG. 8A. The controller(s) 836 may beused for a variety of functions. The controller(s) 836 may be coupled toany of the various other components and systems of the vehicle 800, andmay be used for control of the vehicle 800, artificial intelligence ofthe vehicle 800, infotainment for the vehicle 800, and/or the like.

The vehicle 800 may include a system(s) on a chip (SoC) 804. The SoC 804may include CPU(s) 806, GPU(s) 808, processor(s) 810, cache(s) 812,accelerator(s) 814, data store(s) 816, and/or other components andfeatures not illustrated. The SoC(s) 804 may be used to control thevehicle 800 in a variety of platforms and systems. For example, theSoC(s) 804 may be combined in a system (e.g., the system of the vehicle800) with an HD map 822 which may obtain map refreshes and/or updatesvia a network interface 824 from one or more servers (e.g., server(s)878 of FIG. 8D).

The CPU(s) 806 may include a CPU cluster or CPU complex (alternativelyreferred to herein as a “CCPLEX”). The CPU(s) 806 may include multiplecores and/or L2 caches. For example, in some embodiments, the CPU(s) 806may include eight cores in a coherent multi-processor configuration. Insome embodiments, the CPU(s) 806 may include four dual-core clusterswhere each cluster has a dedicated L2 cache (e.g., a 2 MB L2 cache). TheCPU(s) 806 (e.g., the CCPLEX) may be configured to support simultaneouscluster operation enabling any combination of the clusters of the CPU(s)806 to be active at any given time.

The CPU(s) 806 may implement power management capabilities that includeone or more of the following features: individual hardware blocks may beclock-gated automatically when idle to save dynamic power; each coreclock may be gated when the core is not actively executing instructionsdue to execution of WFI/WFE instructions; each core may be independentlypower-gated; each core cluster may be independently clock-gated when allcores are clock-gated or power-gated; and/or each core cluster may beindependently power-gated when all cores are power-gated. The CPU(s) 806may further implement an enhanced algorithm for managing power states,where allowed power states and expected wakeup times are specified, andthe hardware/microcode determines the best power state to enter for thecore, cluster, and CCPLEX. The processing cores may support simplifiedpower state entry sequences in software with the work offloaded tomicrocode.

The GPU(s) 808 may include an integrated GPU (alternatively referred toherein as an “iGPU”). The GPU(s) 808 may be programmable and may beefficient for parallel workloads. The GPU(s) 808, in some examples, mayuse an enhanced tensor instruction set. The GPU(s) 808 may include oneor more streaming microprocessors, where each streaming microprocessormay include an L1 cache (e.g., an L1 cache with at least 96 KB storagecapacity), and two or more of the streaming microprocessors may share anL2 cache (e.g., an L2 cache with a 512 KB storage capacity). In someembodiments, the GPU(s) 808 may include at least eight streamingmicroprocessors. The GPU(s) 808 may use compute application programminginterface(s) (API(s)). In addition, the GPU(s) 808 may use one or moreparallel computing platforms and/or programming models (e.g., NVIDIA'sCUDA).

The GPU(s) 808 may be power-optimized for best performance in automotiveand embedded use cases. For example, the GPU(s) 808 may be fabricated ona Fin field-effect transistor (FinFET). However, this is not intended tobe limiting and the GPU(s) 808 may be fabricated using othersemiconductor manufacturing processes. Each streaming microprocessor mayincorporate a number of mixed-precision processing cores partitionedinto multiple blocks. For example, and without limitation, 64 PF32 coresand 32 PF64 cores may be partitioned into four processing blocks. Insuch an example, each processing block may be allocated 16 FP32 cores, 8FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs fordeep learning matrix arithmetic, an L0 instruction cache, a warpscheduler, a dispatch unit, and/or a 64 KB register file. In addition,the streaming microprocessors may include independent parallel integerand floating-point data paths to provide for efficient execution ofworkloads with a mix of computation and addressing calculations. Thestreaming microprocessors may include independent thread schedulingcapability to enable finer-grain synchronization and cooperation betweenparallel threads. The streaming microprocessors may include a combinedL1 data cache and shared memory unit in order to improve performancewhile simplifying programming.

The GPU(s) 808 may include a high bandwidth memory (HBM) and/or a 16 GBHBM2 memory subsystem to provide, in some examples, about 900 GB/secondpeak memory bandwidth. In some examples, in addition to, oralternatively from, the HBM memory, a synchronous graphics random-accessmemory (SGRAM) may be used, such as a graphics double data rate typefive synchronous random-access memory (GDDR5).

The GPU(s) 808 may include unified memory technology including accesscounters to allow for more accurate migration of memory pages to theprocessor that accesses them most frequently, thereby improvingefficiency for memory ranges shared between processors. In someexamples, address translation services (ATS) support may be used toallow the GPU(s) 808 to access the CPU(s) 806 page tables directly. Insuch examples, when the GPU(s) 808 memory management unit (MMU)experiences a miss, an address translation request may be transmitted tothe CPU(s) 806. In response, the CPU(s) 806 may look in its page tablesfor the virtual-to-physical mapping for the address and transmits thetranslation back to the GPU(s) 808. As such, unified memory technologymay allow a single unified virtual address space for memory of both theCPU(s) 806 and the GPU(s) 808, thereby simplifying the GPU(s) 808programming and porting of applications to the GPU(s) 808.

In addition, the GPU(s) 808 may include an access counter that may keeptrack of the frequency of access of the GPU(s) 808 to memory of otherprocessors. The access counter may help ensure that memory pages aremoved to the physical memory of the processor that is accessing thepages most frequently.

The SoC(s) 804 may include any number of cache(s) 812, including thosedescribed herein. For example, the cache(s) 812 may include an L3 cachethat is available to both the CPU(s) 806 and the GPU(s) 808 (e.g., thatis connected both the CPU(s) 806 and the GPU(s) 808). The cache(s) 812may include a write-back cache that may keep track of states of lines,such as by using a cache coherence protocol (e.g., MEI, MESI, MSI,etc.). The L3 cache may include 4 MB or more, depending on theembodiment, although smaller cache sizes may be used.

The SoC(s) 804 may include one or more accelerators 814 (e.g., hardwareaccelerators, software accelerators, or a combination thereof). Forexample, the SoC(s) 804 may include a hardware acceleration cluster thatmay include optimized hardware accelerators and/or large on-chip memory.The large on-chip memory (e.g., 4 MB of SRAM), may enable the hardwareacceleration cluster to accelerate neural networks and othercalculations. The hardware acceleration cluster may be used tocomplement the GPU(s) 808 and to off-load some of the tasks of theGPU(s) 808 (e.g., to free up more cycles of the GPU(s) 808 forperforming other tasks). As an example, the accelerator(s) 814 may beused for targeted workloads (e.g., perception, convolutional neuralnetworks (CNNs), etc.) that are stable enough to be amenable toacceleration. The term “CNN,” as used herein, may include all types ofCNNs, including region-based or regional convolutional neural networks(RCNNs) and Fast RCNNs (e.g., as used for object detection).

The accelerator(s) 814 (e.g., the hardware acceleration cluster) mayinclude a deep learning accelerator(s) (DLA). The DLA(s) may include oneor more Tensor processing units (TPUs) that may be configured to providean additional ten trillion operations per second for deep learningapplications and inferencing. The TPUs may be accelerators configuredto, and optimized for, performing image processing functions (e.g., forCNNs, RCNNs, etc.). The DLA(s) may further be optimized for a specificset of neural network types and floating point operations, as well asinferencing. The design of the DLA(s) may provide more performance permillimeter than a general-purpose GPU, and vastly exceeds theperformance of a CPU. The TPU(s) may perform several functions,including a single-instance convolution function, supporting, forexample, INT8, INT16, and FP16 data types for both features and weights,as well as post-processor functions.

The DLA(s) may quickly and efficiently execute neural networks,especially CNNs, on processed or unprocessed data for any of a varietyof functions, including, for example and without limitation: a CNN forobject identification and detection using data from camera sensors; aCNN for distance estimation using data from camera sensors; a CNN foremergency vehicle detection and identification and detection using datafrom microphones; a CNN for facial recognition and vehicle owneridentification using data from camera sensors; and/or a CNN for securityand/or safety related events.

The DLA(s) may perform any function of the GPU(s) 808, and by using aninference accelerator, for example, a designer may target either theDLA(s) or the GPU(s) 808 for any function. For example, the designer mayfocus processing of CNNs and floating point operations on the DLA(s) andleave other functions to the GPU(s) 808 and/or other accelerator(s) 814.

The accelerator(s) 814 (e.g., the hardware acceleration cluster) mayinclude a programmable vision accelerator(s) (PVA), which mayalternatively be referred to herein as a computer vision accelerator.The PVA(s) may be designed and configured to accelerate computer visionalgorithms for the advanced driver assistance systems (ADAS), autonomousdriving, and/or augmented reality (AR) and/or virtual reality (VR)applications. The PVA(s) may provide a balance between performance andflexibility. For example, each PVA(s) may include, for example andwithout limitation, any number of reduced instruction set computer(RISC) cores, direct memory access (DMA), and/or any number of vectorprocessors.

The RISC cores may interact with image sensors (e.g., the image sensorsof any of the cameras described herein), image signal processor(s),and/or the like. Each of the RISC cores may include any amount ofmemory. The RISC cores may use any of a number of protocols, dependingon the embodiment. In some examples, the RISC cores may execute areal-time operating system (RTOS). The RISC cores may be implementedusing one or more integrated circuit devices, application specificintegrated circuits (ASICs), and/or memory devices. For example, theRISC cores may include an instruction cache and/or a tightly coupledRAM.

The DMA may enable components of the PVA(s) to access the system memoryindependently of the CPU(s) 806. The DMA may support any number offeatures used to provide optimization to the PVA including, but notlimited to, supporting multi-dimensional addressing and/or circularaddressing. In some examples, the DMA may support up to six or moredimensions of addressing, which may include block width, block height,block depth, horizontal block stepping, vertical block stepping, and/ordepth stepping.

The vector processors may be programmable processors that may bedesigned to efficiently and flexibly execute programming for computervision algorithms and provide signal processing capabilities. In someexamples, the PVA may include a PVA core and two vector processingsubsystem partitions. The PVA core may include a processor subsystem,DMA engine(s) (e.g., two DMA engines), and/or other peripherals. Thevector processing subsystem may operate as the primary processing engineof the PVA, and may include a vector processing unit (VPU), aninstruction cache, and/or vector memory (e.g., VMEM). A VPU core mayinclude a digital signal processor such as, for example, a singleinstruction, multiple data (SIMD), very long instruction word (VLIW)digital signal processor. The combination of the SIMD and VLIW mayenhance throughput and speed.

Each of the vector processors may include an instruction cache and maybe coupled to dedicated memory. As a result, in some examples, each ofthe vector processors may be configured to execute independently of theother vector processors. In other examples, the vector processors thatare included in a particular PVA may be configured to employ dataparallelism. For example, in some embodiments, the plurality of vectorprocessors included in a single PVA may execute the same computer visionalgorithm, but on different regions of an image. In other examples, thevector processors included in a particular PVA may simultaneouslyexecute different computer vision algorithms, on the same image, or evenexecute different algorithms on sequential images or portions of animage. Among other things, any number of PVAs may be included in thehardware acceleration cluster and any number of vector processors may beincluded in each of the PVAs. In addition, the PVA(s) may includeadditional error correcting code (ECC) memory, to enhance overall systemsafety.

The accelerator(s) 814 (e.g., the hardware acceleration cluster) mayinclude a computer vision network on-chip and SRAM, for providing ahigh-bandwidth, low latency SRAM for the accelerator(s) 814. In someexamples, the on-chip memory may include at least 4 MB SRAM, consistingof, for example and without limitation, eight field-configurable memoryblocks, that may be accessible by both the PVA and the DLA. Each pair ofmemory blocks may include an advanced peripheral bus (APB) interface,configuration circuitry, a controller, and a multiplexer. Any type ofmemory may be used. The PVA and DLA may access the memory via a backbonethat provides the PVA and DLA with high-speed access to memory. Thebackbone may include a computer vision network on-chip thatinterconnects the PVA and the DLA to the memory (e.g., using the APB).

The computer vision network on-chip may include an interface thatdetermines, before transmission of any control signal/address/data, thatboth the PVA and the DLA provide ready and valid signals. Such aninterface may provide for separate phases and separate channels fortransmitting control signals/addresses/data, as well as burst-typecommunications for continuous data transfer. This type of interface maycomply with ISO 26262 or IEC 61508 standards, although other standardsand protocols may be used.

In some examples, the SoC(s) 804 may include a real-time ray-tracinghardware accelerator, such as described in U.S. patent application Ser.No. 16/101,232, filed on Aug. 10, 2018. The real-time ray-tracinghardware accelerator may be used to quickly and efficiently determinethe positions and extents of objects (e.g., within a world model), togenerate real0time visualization simulations, for RADAR signalinterpretation, for sound propagation synthesis and/or analysis, forsimulation of SONAR systems, for general wave propagation simulation,for comparison to LIDAR data for purposes of localization and/or otherfunctions, and/or for other uses.

The accelerator(s) 814 (e.g., the hardware accelerator cluster) have awide array of uses for autonomous driving. The PVA may be a programmablevision accelerator that may be used for key processing stages in ADASand autonomous vehicles. The PVA's capabilities are a good match foralgorithmic domains needing predictable processing, at low power and lowlatency. In other words, the PVA performs well on semi-dense or denseregular computation, even on small data sets, which need predictablerun-times with low latency and low power. Thus, in the context ofplatforms for autonomous vehicles, the PVAs are designed to run classiccomputer vision algorithms, as they are efficient at object detectionand operating on integer math.

For example, according to one embodiment of the technology, the PVA isused to perform computer stereo vision. A semi-global matching-basedalgorithm may be used in some examples, although this is not intended tobe limiting. Many applications for Level 3-5 autonomous driving requiremotion estimation/stereo matching on-the-fly (e.g., structure frommotion, pedestrian recognition, lane detection, etc.). The PVA mayperform computer stereo vision function on inputs from two monocularcameras.

In some examples, the PVA may be used to perform dense optical flow.According to process raw RADAR data (e.g., using a 4D Fast FourierTransform) to provide Processed RADAR. In other examples, the PVA isused for time of flight depth processing, by processing raw time offlight data to provide processed time of flight data, for example.

The DLA may be used to run any type of network to enhance control anddriving safety, including for example, a neural network that outputs ameasure of confidence for each object detection. Such a confidence valuemay be interpreted as a probability, or as providing a relative “weight”of each detection compared to other detections. This confidence valueenables the system to make further decisions regarding which detectionsshould be considered as true positive detections rather than falsepositive detections. For example, the system may set a threshold valuefor the confidence and consider only the detections exceeding thethreshold value as true positive detections. In an automatic emergencybraking (AEB) system, false positive detections would cause the vehicleto automatically perform emergency braking, which is obviouslyundesirable. Therefore, only the most confident detections should beconsidered as triggers for AEB. The DLA may run a neural network forregressing the confidence value. The neural network may take as itsinput at least some subset of parameters, such as bounding boxdimensions, ground plane estimate obtained (e.g. from anothersubsystem), inertial measurement unit (IMU) sensor 866 output thatcorrelates with the vehicle 800 orientation, distance, 3D locationestimates of the object obtained from the neural network and/or othersensors (e.g., LIDAR sensor(s) 864 or RADAR sensor(s) 860), amongothers.

The SoC(s) 804 may include data store(s) 816 (e.g., memory). The datastore(s) 816 may be on-chip memory of the SoC(s) 804, which may storeneural networks to be executed on the GPU and/or the DLA. In someexamples, the data store(s) 816 may be large enough in capacity to storemultiple instances of neural networks for redundancy and safety. Thedata store(s) 812 may comprise L2 or L3 cache(s) 812. Reference to thedata store(s) 816 may include reference to the memory associated withthe PVA, DLA, and/or other accelerator(s) 814, as described herein.

The SoC(s) 804 may include one or more processor(s) 810 (e.g., embeddedprocessors).

The processor(s) 810 may include a boot and power management processorthat may be a dedicated processor and subsystem to handle boot power andmanagement functions and related security enforcement. The boot andpower management processor may be a part of the SoC(s) 804 boot sequenceand may provide runtime power management services. The boot power andmanagement processor may provide clock and voltage programming,assistance in system low power state transitions, management of SoC(s)804 thermals and temperature sensors, and/or management of the SoC(s)804 power states. Each temperature sensor may be implemented as aring-oscillator whose output frequency is proportional to temperature,and the SoC(s) 804 may use the ring-oscillators to detect temperaturesof the CPU(s) 806, GPU(s) 808, and/or accelerator(s) 814. Iftemperatures are determined to exceed a threshold, the boot and powermanagement processor may enter a temperature fault routine and put theSoC(s) 804 into a lower power state and/or put the vehicle 800 into achauffeur to safe stop mode (e.g., bring the vehicle 800 to a safestop).

The processor(s) 810 may further include a set of embedded processorsthat may serve as an audio processing engine. The audio processingengine may be an audio subsystem that enables full hardware support formulti-channel audio over multiple interfaces, and a broad and flexiblerange of audio I/O interfaces. In some examples, the audio processingengine is a dedicated processor core with a digital signal processorwith dedicated RAM.

The processor(s) 810 may further include an always on processor enginethat may provide necessary hardware features to support low power sensormanagement and wake use cases. The always on processor engine mayinclude a processor core, a tightly coupled RAM, supporting peripherals(e.g., timers and interrupt controllers), various I/O controllerperipherals, and routing logic.

The processor(s) 810 may further include a safety cluster engine thatincludes a dedicated processor subsystem to handle safety management forautomotive applications. The safety cluster engine may include two ormore processor cores, a tightly coupled RAM, support peripherals (e.g.,timers, an interrupt controller, etc.), and/or routing logic. In asafety mode, the two or more cores may operate in a lockstep mode andfunction as a single core with comparison logic to detect anydifferences between their operations.

The processor(s) 810 may further include a real-time camera engine thatmay include a dedicated processor subsystem for handling real-timecamera management.

The processor(s) 810 may further include a high-dynamic range signalprocessor that may include an image signal processor that is a hardwareengine that is part of the camera processing pipeline.

The processor(s) 810 may include a video image compositor that may be aprocessing block (e.g., implemented on a microprocessor) that implementsvideo post-processing functions needed by a video playback applicationto produce the final image for the player window. The video imagecompositor may perform lens distortion correction on wide-view camera(s)870, surround camera(s) 874, and/or on in-cabin monitoring camerasensors. In-cabin monitoring camera sensor is preferably monitored by aneural network running on another instance of the Advanced SoC,configured to identify in cabin events and respond accordingly. Anin-cabin system may perform lip reading to activate cellular service andplace a phone call, dictate emails, change the vehicle's destination,activate or change the vehicle's infotainment system and settings, orprovide voice-activated web surfing. Certain functions are available tothe driver only when the vehicle is operating in an autonomous mode, andare disabled otherwise.

The video image compositor may include enhanced temporal noise reductionfor both spatial and temporal noise reduction. For example, where motionoccurs in a video, the noise reduction weights spatial informationappropriately, decreasing the weight of information provided by adjacentframes. Where an image or portion of an image does not include motion,the temporal noise reduction performed by the video image compositor mayuse information from the previous image to reduce noise in the currentimage.

The video image compositor may also be configured to perform stereorectification on input stereo lens frames. The video image compositormay further be used for user interface composition when the operatingsystem desktop is in use, and the GPU(s) 808 is not required tocontinuously render new surfaces. Even when the GPU(s) 808 is powered onand active doing 3D rendering, the video image compositor may be used tooffload the GPU(s) 808 to improve performance and responsiveness.

The SoC(s) 804 may further include a mobile industry processor interface(MIPI) camera serial interface for receiving video and input fromcameras, a high-speed interface, and/or a video input block that may beused for camera and related pixel input functions. The SoC(s) 804 mayfurther include an input/output controller(s) that may be controlled bysoftware and may be used for receiving I/O signals that are uncommittedto a specific role.

The SoC(s) 804 may further include a broad range of peripheralinterfaces to enable communication with peripherals, audio codecs, powermanagement, and/or other devices. The SoC(s) 804 may be used to processdata from cameras (e.g., connected over Gigabit Multimedia Serial Linkand Ethernet), sensors (e.g., LIDAR sensor(s) 864, RADAR sensor(s) 860,etc. that may be connected over Ethernet), data from bus 802 (e.g.,speed of vehicle 800, steering wheel position, etc.), data from GNSSsensor(s) 858 (e.g., connected over Ethernet or CAN bus). The SoC(s) 804may further include dedicated high-performance mass storage controllersthat may include their own DMA engines, and that may be used to free theCPU(s) 806 from routine data management tasks.

The SoC(s) 804 may be an end-to-end platform with a flexiblearchitecture that spans automation levels 3-5, thereby providing acomprehensive functional safety architecture that leverages and makesefficient use of computer vision and ADAS techniques for diversity andredundancy, provides a platform for a flexible, reliable drivingsoftware stack, along with deep learning tools. The SoC(s) 804 may befaster, more reliable, and even more energy-efficient andspace-efficient than conventional systems. For example, theaccelerator(s) 814, when combined with the CPU(s) 806, the GPU(s) 808,and the data store(s) 816, may provide for a fast, efficient platformfor level 3-5 autonomous vehicles.

The technology thus provides capabilities and functionality that cannotbe achieved by conventional systems. For example, computer visionalgorithms may be executed on CPUs, which may be configured usinghigh-level programming language, such as the C programming language, toexecute a wide variety of processing algorithms across a wide variety ofvisual data. However, CPUs are oftentimes unable to meet the performancerequirements of many computer vision applications, such as those relatedto execution time and power consumption, for example. In particular,many CPUs are unable to execute complex object detection algorithms inreal-time, which is a requirement of in-vehicle ADAS applications, and arequirement for practical Level 3-5 autonomous vehicles.

In contrast to conventional systems, by providing a CPU complex, GPUcomplex, and a hardware acceleration cluster, the technology describedherein allows for multiple neural networks to be performedsimultaneously and/or sequentially, and for the results to be combinedtogether to enable Level 3-5 autonomous driving functionality. Forexample, a CNN executing on the DLA or dGPU (e.g., the GPU(s) 820) mayinclude a text and word recognition, allowing the supercomputer to readand understand traffic signs, including signs for which the neuralnetwork has not been specifically trained. The DLA may further include aneural network that is able to identify, interpret, and providessemantic understanding of the sign, and to pass that semanticunderstanding to the path planning modules running on the CPU Complex.

As another example, multiple neural networks may be run simultaneously,as is required for Level 3, 4, or 5 driving. For example, a warning signconsisting of “Caution: flashing lights indicate icy conditions,” alongwith an electric light, may be independently or collectively interpretedby several neural networks. The sign itself may be identified as atraffic sign by a first deployed neural network (e.g., a neural networkthat has been trained), the text “Flashing lights indicate icyconditions” may be interpreted by a second deployed neural network,which informs the vehicle's path planning software (preferably executingon the CPU Complex) that when flashing lights are detected, icyconditions exist. The flashing light may be identified by operating athird deployed neural network over multiple frames, informing thevehicle's path-planning software of the presence (or absence) offlashing lights. All three neural networks may run simultaneously, suchas within the DLA and/or on the GPU(s) 808.

In some examples, a CNN for facial recognition and vehicle owneridentification may use data from camera sensors to identify the presenceof an authorized driver and/or owner of the vehicle 800. The always onsensor processing engine may be used to unlock the vehicle when theowner approaches the driver door and turn on the lights, and, insecurity mode, to disable the vehicle when the owner leaves the vehicle.In this way, the SoC(s) 804 provide for security against theft and/orcarjacking.

In another example, a CNN for emergency vehicle detection andidentification may use data from microphones 896 to detect and identifyemergency vehicle sirens. In contrast to conventional systems, that usegeneral classifiers to detect sirens and manually extract features, theSoC(s) 804 use the CNN for classifying environmental and urban sounds,as well as classifying visual data. In a preferred embodiment, the CNNrunning on the DLA is trained to identify the relative closing speed ofthe emergency vehicle (e.g., by using the Doppler effect). The CNN mayalso be trained to identify emergency vehicles specific to the localarea in which the vehicle is operating, as identified by GNSS sensor(s)858. Thus, for example, when operating in Europe the CNN will seek todetect European sirens, and when in the United States the CNN will seekto identify only North American sirens. Once an emergency vehicle isdetected, a control program may be used to execute an emergency vehiclesafety routine, slowing the vehicle, pulling over to the side of theroad, parking the vehicle, and/or idling the vehicle, with theassistance of ultrasonic sensors 862, until the emergency vehicle(s)passes.

The vehicle may include a CPU(s) 818 (e.g., discrete CPU(s), ordCPU(s)), that may be coupled to the SoC(s) 804 via a high-speedinterconnect (e.g., PCIe). The CPU(s) 818 may include an X86 processor,for example. The CPU(s) 818 may be used to perform any of a variety offunctions, including arbitrating potentially inconsistent resultsbetween ADAS sensors and the SoC(s) 804, and/or monitoring the statusand health of the controller(s) 836 and/or infotainment SoC 830, forexample.

The vehicle 800 may include a GPU(s) 820 (e.g., discrete GPU(s), ordGPU(s)), that may be coupled to the SoC(s) 804 via a high-speedinterconnect (e.g., NVIDIA's NVLINK). The GPU(s) 820 may provideadditional artificial intelligence functionality, such as by executingredundant and/or different neural networks, and may be used to trainand/or update neural networks based on input (e.g., sensor data) fromsensors of the vehicle 800.

The vehicle 800 may further include the network interface 824 which mayinclude one or more wireless antennas 826 (e.g., one or more wirelessantennas for different communication protocols, such as a cellularantenna, a Bluetooth antenna, etc.). The network interface 824 may beused to enable wireless connectivity over the Internet with the cloud(e.g., with the server(s) 878 and/or other network devices), with othervehicles, and/or with computing devices (e.g., client devices ofpassengers). To communicate with other vehicles, a direct link may beestablished between the two vehicles and/or an indirect link may beestablished (e.g., across networks and over the Internet). Direct linksmay be provided using a vehicle-to-vehicle communication link. Thevehicle-to-vehicle communication link may provide the vehicle 800information about vehicles in proximity to the vehicle 800 (e.g.,vehicles in front of, on the side of, and/or behind the vehicle 800).This functionality may be part of a cooperative adaptive cruise controlfunctionality of the vehicle 800.

The network interface 824 may include a SoC that provides modulation anddemodulation functionality and enables the controller(s) 836 tocommunicate over wireless networks. The network interface 824 mayinclude a radio frequency front-end for up-conversion from baseband toradio frequency, and down conversion from radio frequency to baseband.The frequency conversions may be performed through well-known processes,and/or may be performed using super-heterodyne processes. In someexamples, the radio frequency front end functionality may be provided bya separate chip. The network interface may include wirelessfunctionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000,Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and/or otherwireless protocols.

The vehicle 800 may further include data store(s) 828 which may includeoff-chip (e.g., off the SoC(s) 804) storage. The data store(s) 828 mayinclude one or more storage elements including RAM, SRAM, DRAM, VRAM,Flash, hard disks, and/or other components and/or devices that may storeat least one bit of data.

The vehicle 800 may further include GNSS sensor(s) 858. The GNSSsensor(s) 858 (e.g., GPS and/or assisted GPS sensors), to assist inmapping, perception, occupancy grid generation, and/or path planningfunctions. Any number of GNSS sensor(s) 858 may be used, including, forexample and without limitation, a GPS using a USB connector with anEthernet to Serial (RS-232) bridge.

The vehicle 800 may further include RADAR sensor(s) 860. The RADARsensor(s) 860 may be used by the vehicle 800 for long-range vehicledetection, even in darkness and/or severe weather conditions. RADARfunctional safety levels may be ASIL B. The RADAR sensor(s) 860 may usethe CAN and/or the bus 802 (e.g., to transmit data generated by theRADAR sensor(s) 860) for control and to access object tracking data,with access to Ethernet to access raw data in some examples. A widevariety of RADAR sensor types may be used. For example, and withoutlimitation, the RADAR sensor(s) 860 may be suitable for front, rear, andside RADAR use. In some example, Pulse Doppler RADAR sensor(s) are used.

The RADAR sensor(s) 860 may include different configurations, such aslong range with narrow field of view, short range with wide field ofview, short range side coverage, etc. In some examples, long-range RADARmay be used for adaptive cruise control functionality. The long-rangeRADAR systems may provide a broad field of view realized by two or moreindependent scans, such as within a 250 m range. The RADAR sensor(s) 860may help in distinguishing between static and moving objects, and may beused by ADAS systems for emergency brake assist and forward collisionwarning. Long-range RADAR sensors may include monostatic multimodalRADAR with multiple (e.g., six or more) fixed RADAR antennae and ahigh-speed CAN and FlexRay interface. In an example with six antennae,the central four antennae may create a focused beam pattern, designed torecord the vehicle's 800 surroundings at higher speeds with minimalinterference from traffic in adjacent lanes. The other two antennae mayexpand the field of view, making it possible to quickly detect vehiclesentering or leaving the vehicle's 800 lane.

Mid-range RADAR systems may include, as an example, a range of up to 860m (front) or 80 m (rear), and a field of view of up to 42 degrees(front) or 850 degrees (rear). Short-range RADAR systems may include,without limitation, RADAR sensors designed to be installed at both endsof the rear bumper. When installed at both ends of the rear bumper, sucha RADAR sensor systems may create two beams that constantly monitor theblind spot in the rear and next to the vehicle.

Short-range RADAR systems may be used in an ADAS system for blind spotdetection and/or lane change assist.

The vehicle 800 may further include ultrasonic sensor(s) 862. Theultrasonic sensor(s) 862, which may be positioned at the front, back,and/or the sides of the vehicle 800, may be used for park assist and/orto create and update an occupancy grid. A wide variety of ultrasonicsensor(s) 862 may be used, and different ultrasonic sensor(s) 862 may beused for different ranges of detection (e.g., 2.5 m, 4 m). Theultrasonic sensor(s) 862 may operate at functional safety levels of ASILB.

The vehicle 800 may include LIDAR sensor(s) 864. The LIDAR sensor(s) 864may be used for object and pedestrian detection, emergency braking,collision avoidance, and/or other functions. The LIDAR sensor(s) 864 maybe functional safety level ASIL B. In some examples, the vehicle 800 mayinclude multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that mayuse Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

In some examples, the LIDAR sensor(s) 864 may be capable of providing alist of objects and their distances for a 360-degree field of view.Commercially available LIDAR sensor(s) 864 may have an advertised rangeof approximately 800 m, with an accuracy of 2 cm-3 cm, and with supportfor an 800 Mbps Ethernet connection, for example. In some examples, oneor more non-protruding LIDAR sensors 864 may be used. In such examples,the LIDAR sensor(s) 864 may be implemented as a small device that may beembedded into the front, rear, sides, and/or corners of the vehicle 800.The LIDAR sensor(s) 864, in such examples, may provide up to an820-degree horizontal and 35-degree vertical field-of-view, with a 200 mrange even for low-reflectivity objects. Front-mounted LIDAR sensor(s)864 may be configured for a horizontal field of view between 45 degreesand 135 degrees.

In some examples, LIDAR technologies, such as 3D flash LIDAR, may alsobe used. 3D Flash LIDAR uses a flash of a laser as a transmissionsource, to illuminate vehicle surroundings up to approximately 200 m. Aflash LIDAR unit includes a receptor, which records the laser pulsetransit time and the reflected light on each pixel, which in turncorresponds to the range from the vehicle to the objects. Flash LIDARmay allow for highly accurate and distortion-free images of thesurroundings to be generated with every laser flash. In some examples,four flash LIDAR sensors may be deployed, one at each side of thevehicle 800. Available 3D flash LIDAR systems include a solid-state 3Dstaring array LIDAR camera with no moving parts other than a fan (e.g.,a non-scanning LIDAR device). The flash LIDAR device may use a 5nanosecond class I (eye-safe) laser pulse per frame and may capture thereflected laser light in the form of 3D range point clouds andco-registered intensity data. By using flash LIDAR, and because flashLIDAR is a solid-state device with no moving parts, the LIDAR sensor(s)864 may be less susceptible to motion blur, vibration, and/or shock.

The vehicle may further include IMU sensor(s) 866. The IMU sensor(s) 866may be located at a center of the rear axle of the vehicle 800, in someexamples. The IMU sensor(s) 866 may include, for example and withoutlimitation, an accelerometer(s), a magnetometer(s), a gyroscope(s), amagnetic compass(es), and/or other sensor types. In some examples, suchas in six-axis applications, the IMU sensor(s) 866 may includeaccelerometers and gyroscopes, while in nine-axis applications, the IMUsensor(s) 866 may include accelerometers, gyroscopes, and magnetometers.

In some embodiments, the IMU sensor(s) 866 may be implemented as aminiature, high performance GPS-Aided Inertial Navigation System(GPS/INS) that combines micro-electro-mechanical systems (MEMS) inertialsensors, a high-sensitivity GPS receiver, and advanced Kalman filteringalgorithms to provide estimates of position, velocity, and attitude. Assuch, in some examples, the IMU sensor(s) 866 may enable the vehicle 800to estimate heading without requiring input from a magnetic sensor bydirectly observing and correlating the changes in velocity from GPS tothe IMU sensor(s) 866. In some examples, the IMU sensor(s) 866 and theGNSS sensor(s) 858 may be combined in a single integrated unit.

The vehicle may include microphone(s) 896 placed in and/or around thevehicle 800. The microphone(s) 896 may be used for emergency vehicledetection and identification, among other things.

The vehicle may further include any number of camera types, includingstereo camera(s) 868, wide-view camera(s) 870, infrared camera(s) 872,surround camera(s) 874, long-range and/or mid-range camera(s) 898,and/or other camera types. The cameras may be used to capture image dataaround an entire periphery of the vehicle 800. The types of cameras useddepends on the embodiments and requirements for the vehicle 800, and anycombination of camera types may be used to provide the necessarycoverage around the vehicle 800. In addition, the number of cameras maydiffer depending on the embodiment. For example, the vehicle may includesix cameras, seven cameras, ten cameras, twelve cameras, and/or anothernumber of cameras. The cameras may support, as an example and withoutlimitation, Gigabit Multimedia Serial Link (GMSL) and/or GigabitEthernet. Each of the camera(s) is described with more detail hereinwith respect to FIG. 8A and FIG. 8B.

The vehicle 800 may further include vibration sensor(s) 842. Thevibration sensor(s) 842 may measure vibrations of components of thevehicle, such as the axle(s). For example, changes in vibrations mayindicate a change in road surfaces. In another example, when two or morevibration sensors 842 are used, the differences between the vibrationsmay be used to determine friction or slippage of the road surface (e.g.,when the difference in vibration is between a power-driven axle and afreely rotating axle).

The vehicle 800 may include an ADAS system 838. The ADAS system 838 mayinclude a SoC, in some examples. The ADAS system 838 may includeautonomous/adaptive/automatic cruise control (ACC), cooperative adaptivecruise control (CACC), forward crash warning (FCW), automatic emergencybraking (AEB), lane departure warnings (LDW), lane keep assist (LKA),blind spot warning (BSW), rear cross-traffic warning (RCTW), collisionwarning systems (CWS), lane centering (LC), and/or other features andfunctionality.

The ACC systems may use RADAR sensor(s) 860, LIDAR sensor(s) 864, and/ora camera(s). The ACC systems may include longitudinal ACC and/or lateralACC. Longitudinal ACC monitors and controls the distance to the vehicleimmediately ahead of the vehicle 800 and automatically adjust thevehicle speed to maintain a safe distance from vehicles ahead. LateralACC performs distance keeping, and advises the vehicle 800 to changelanes when necessary. Lateral ACC is related to other ADAS applicationssuch as LCA and CWS.

CACC uses information from other vehicles that may be received via thenetwork interface 824 and/or the wireless antenna(s) 826 from othervehicles via a wireless link, or indirectly, over a network connection(e.g., over the Internet). Direct links may be provided by avehicle-to-vehicle (V2V) communication link, while indirect links may beinfrastructure-to-vehicle (I2V) communication link In general, the V2Vcommunication concept provides information about the immediatelypreceding vehicles (e.g., vehicles immediately ahead of and in the samelane as the vehicle 800), while the I2V communication concept providesinformation about traffic further ahead. CACC systems may include eitheror both I2V and V2V information sources. Given the information of thevehicles ahead of the vehicle 800, CACC may be more reliable and it haspotential to improve traffic flow smoothness and reduce congestion onthe road.

FCW systems are designed to alert the driver to a hazard, so that thedriver may take corrective action. FCW systems use a front-facing cameraand/or RADAR sensor(s) 860, coupled to a dedicated processor, DSP, FPGA,and/or ASIC, that is electrically coupled to driver feedback, such as adisplay, speaker, and/or vibrating component. FCW systems may provide awarning, such as in the form of a sound, visual warning, vibrationand/or a quick brake pulse.

AEB systems detect an impending forward collision with another vehicleor other object, and may automatically apply the brakes if the driverdoes not take corrective action within a specified time or distanceparameter. AEB systems may use front-facing camera(s) and/or RADARsensor(s) 860, coupled to a dedicated processor, DSP, FPGA, and/or ASIC.When the AEB system detects a hazard, it typically first alerts thedriver to take corrective action to avoid the collision and, if thedriver does not take corrective action, the AEB system may automaticallyapply the brakes in an effort to prevent, or at least mitigate, theimpact of the predicted collision. AEB systems, may include techniquessuch as dynamic brake support and/or crash imminent braking.

LDW systems provide visual, audible, and/or tactile warnings, such assteering wheel or seat vibrations, to alert the driver when the vehicle800 crosses lane markings A LDW system does not activate when the driverindicates an intentional lane departure, by activating a turn signal.LDW systems may use front-side facing cameras, coupled to a dedicatedprocessor, DSP, FPGA, and/or ASIC, that is electrically coupled todriver feedback, such as a display, speaker, and/or vibrating component.

LKA systems are a variation of LDW systems. LKA systems provide steeringinput or braking to correct the vehicle 800 if the vehicle 800 starts toexit the lane.

BSW systems detects and warn the driver of vehicles in an automobile'sblind spot. BSW systems may provide a visual, audible, and/or tactilealert to indicate that merging or changing lanes is unsafe. The systemmay provide an additional warning when the driver uses a turn signal.BSW systems may use rear-side facing camera(s) and/or RADAR sensor(s)860, coupled to a dedicated processor, DSP, FPGA, and/or ASIC, that iselectrically coupled to driver feedback, such as a display, speaker,and/or vibrating component.

RCTW systems may provide visual, audible, and/or tactile notificationwhen an object is detected outside the rear-camera range when thevehicle 800 is backing up. Some RCTW systems include AEB to ensure thatthe vehicle brakes are applied to avoid a crash. RCTW systems may useone or more rear-facing RADAR sensor(s) 860, coupled to a dedicatedprocessor, DSP, FPGA, and/or ASIC, that is electrically coupled todriver feedback, such as a display, speaker, and/or vibrating component.

Conventional ADAS systems may be prone to false positive results, whichmay be annoying and distracting to a driver, but typically are notcatastrophic, because the ADAS systems alert the driver and allow thedriver to decide whether a safety condition truly exists and actaccordingly. However, in an autonomous vehicle 800, the vehicle 800itself must, in the case of conflicting results, decide whether to heedthe result from a primary computer or a secondary computer (e.g., afirst controller 836 or a second controller 836). For example, in someembodiments, the ADAS system 838 may be a backup and/or secondarycomputer for providing perception information to a backup computerrationality module. The backup computer rationality monitor may run aredundant diverse software on hardware components to detect faults inperception and dynamic driving tasks. Outputs from the ADAS system 838may be provided to a supervisory MCU. If outputs from the primarycomputer and the secondary computer conflict, the supervisory MCU mustdetermine how to reconcile the conflict to ensure safe operation.

In some examples, the primary computer may be configured to provide thesupervisory MCU with a confidence score, indicating the primarycomputer's confidence in the chosen result. If the confidence scoreexceeds a threshold, the supervisory MCU may follow the primarycomputer's direction, regardless of whether the secondary computerprovides a conflicting or inconsistent result. Where the confidencescore does not meet the threshold, and where the primary and secondarycomputer indicate different results (e.g., the conflict), thesupervisory MCU may arbitrate between the computers to determine theappropriate outcome.

The supervisory MCU may be configured to run a neural network(s) that istrained and configured to determine, based on outputs from the primarycomputer and the secondary computer, conditions under which thesecondary computer provides false alarms. Thus, the neural network(s) inthe supervisory MCU may learn when the secondary computer's output maybe trusted, and when it cannot. For example, when the secondary computeris a RADAR-based FCW system, a neural network(s) in the supervisory MCUmay learn when the FCW system is identifying metallic objects that arenot, in fact, hazards, such as a drainage grate or manhole cover thattriggers an alarm. Similarly, when the secondary computer is acamera-based LDW system, a neural network in the supervisory MCU maylearn to override the LDW when bicyclists or pedestrians are present anda lane departure is, in fact, the safest maneuver. In embodiments thatinclude a neural network(s) running on the supervisory MCU, thesupervisory MCU may include at least one of a DLA or GPU suitable forrunning the neural network(s) with associated memory. In preferredembodiments, the supervisory MCU may comprise and/or be included as acomponent of the SoC(s) 804.

In other examples, ADAS system 838 may include a secondary computer thatperforms ADAS functionality using traditional rules of computer vision.As such, the secondary computer may use classic computer vision rules(if-then), and the presence of a neural network(s) in the supervisoryMCU may improve reliability, safety and performance. For example, thediverse implementation and intentional non-identity makes the overallsystem more fault-tolerant, especially to faults caused by software (orsoftware-hardware interface) functionality. For example, if there is asoftware bug or error in the software running on the primary computer,and the non-identical software code running on the secondary computerprovides the same overall result, the supervisory MCU may have greaterconfidence that the overall result is correct, and the bug in softwareor hardware on primary computer is not causing material error.

In some examples, the output of the ADAS system 838 may be fed into theprimary computer's perception block and/or the primary computer'sdynamic driving task block. For example, if the ADAS system 838indicates a forward crash warning due to an object immediately ahead,the perception block may use this information when identifying objects.In other examples, the secondary computer may have its own neuralnetwork, which is trained and thus reduces the risk of false positives,as described herein.

The vehicle 800 may further include the infotainment SoC 830 (e.g., anin-vehicle infotainment system (IVI)). Although illustrated anddescribed as a SoC, the infotainment system may not be a SoC, and mayinclude two or more discrete components. The infotainment SoC 830 mayinclude a combination of hardware and software that may be used toprovide audio (e.g., music, a personal digital assistant, navigationalinstructions, news, radio, etc.), video (e.g., TV, movies, streaming,etc.), phone (e.g., hands-free calling), network connectivity (e.g.,LTE, WiFi, etc.), and/or information services (e.g., navigation systems,rear-parking assistance, a radio data system, vehicle relatedinformation such as fuel level, total distance covered, brake fuellevel, oil level, door open/close, air filter information, etc.) to thevehicle 800. For example, the infotainment SoC 830 may radios, diskplayers, navigation systems, video players, USB and Bluetoothconnectivity, carputers, in-car entertainment, WiFi, steering wheelaudio controls, hands free voice control, a heads-up display (HUD), anHMI display 834, a telematics device, a control panel (e.g., forcontrolling and/or interacting with various components, features, and/orsystems), and/or other components. The infotainment SoC 830 may furtherbe used to provide information (e.g., visual and/or audible) to auser(s) of the vehicle, such as information from the ADAS system 838,autonomous driving information such as planned vehicle maneuvers,trajectories, surrounding environment information (e.g., intersectioninformation, vehicle information, road information, etc.), and/or otherinformation.

The infotainment SoC 830 may include GPU functionality. The infotainmentSoC 830 may communicate over the bus 802 (e.g., CAN bus, Ethernet, etc.)with other devices, systems, and/or components of the vehicle 800. Insome examples, the infotainment SoC 830 may be coupled to a supervisoryMCU such that the GPU of the infotainment system may perform someself-driving functions in the event that the primary controller(s) 836(e.g., the primary and/or backup computers of the vehicle 800) fail. Insuch an example, the infotainment SoC 830 may put the vehicle 800 into achauffeur to safe stop mode, as described herein.

The vehicle 800 may further include an instrument cluster 832 (e.g., adigital dash, an electronic instrument cluster, a digital instrumentpanel, etc.). The instrument cluster 832 may include a controller and/orsupercomputer (e.g., a discrete controller or supercomputer). Theinstrument cluster 832 may include a set of instrumentation such as aspeedometer, fuel level, oil pressure, tachometer, odometer, turnindicators, gearshift position indicator, seat belt warning light(s),parking-brake warning light(s), engine-malfunction light(s), airbag(SRS) system information, lighting controls, safety system controls,navigation information, etc. In some examples, information may bedisplayed and/or shared among the infotainment SoC 830 and theinstrument cluster 832. In other words, the instrument cluster 832 maybe included as part of the infotainment SoC 830, or vice versa.

FIG. 8D is a system diagram for communication between cloud-basedserver(s) and the example autonomous vehicle 800 of FIG. 8A, inaccordance with some embodiments of the present disclosure. The system876 may include server(s) 878, network(s) 890, and vehicles, includingthe vehicle 800. The server(s) 878 may include a plurality of GPUs884(A)-884(H) (collectively referred to herein as GPUs 884), PCIeswitches 882(A)-882(H) (collectively referred to herein as PCIe switches882), and/or CPUs 880(A)-880(B) (collectively referred to herein as CPUs880). The GPUs 884, the CPUs 880, and the PCIe switches may beinterconnected with high-speed interconnects such as, for example andwithout limitation, NVLink interfaces 888 developed by NVIDIA and/orPCIe connections 886. In some examples, the GPUs 884 are connected viaNVLink and/or NVSwitch SoC and the GPUs 884 and the PCIe switches 882are connected via PCIe interconnects. Although eight GPUs 884, two CPUs880, and two PCIe switches are illustrated, this is not intended to belimiting. Depending on the embodiment, each of the server(s) 878 mayinclude any number of GPUs 884, CPUs 880, and/or PCIe switches. Forexample, the server(s) 878 may each include eight, sixteen, thirty-two,and/or more GPUs 884.

The server(s) 878 may receive, over the network(s) 890 and from thevehicles, image data representative of images showing unexpected orchanged road conditions, such as recently commenced road-work. Theserver(s) 878 may transmit, over the network(s) 890 and to the vehicles,neural networks 892, updated neural networks 892, and/or map information894, including information regarding traffic and road conditions. Theupdates to the map information 894 may include updates for the HD map822, such as information regarding construction sites, potholes,detours, flooding, and/or other obstructions. In some examples, theneural networks 892, the updated neural networks 892, and/or the mapinformation 894 may have resulted from new training and/or experiencesrepresented in data received from any number of vehicles in theenvironment, and/or based on training performed at a datacenter (e.g.,using the server(s) 878 and/or other servers).

The server(s) 878 may be used to train machine learning models (e.g.,neural networks) based on training data. The training data may begenerated by the vehicles, and/or may be generated in a simulation(e.g., using a game engine). In some examples, the training data istagged (e.g., where the neural network benefits from supervisedlearning) and/or undergoes other pre-processing, while in other examplesthe training data is not tagged and/or pre-processed (e.g., where theneural network does not require supervised learning). Once the machinelearning models are trained, the machine learning models may be used bythe vehicles (e.g., transmitted to the vehicles over the network(s) 890,and/or the machine learning models may be used by the server(s) 878 toremotely monitor the vehicles.

In some examples, the server(s) 878 may receive data from the vehiclesand apply the data to up-to-date real-time neural networks for real-timeintelligent inferencing. The server(s) 878 may include deep-learningsupercomputers and/or dedicated AI computers powered by GPU(s) 884, suchas a DGX and DGX Station machines developed by NVIDIA. However, in someexamples, the server(s) 878 may include deep learning infrastructurethat use only CPU-powered datacenters.

The deep-learning infrastructure of the server(s) 878 may be capable offast, real-time inferencing, and may use that capability to evaluate andverify the health of the processors, software, and/or associatedhardware in the vehicle 800. For example, the deep-learninginfrastructure may receive periodic updates from the vehicle 800, suchas a sequence of images and/or objects that the vehicle 800 has locatedin that sequence of images (e.g., via computer vision and/or othermachine learning object classification techniques). The deep-learninginfrastructure may run its own neural network to identify the objectsand compare them with the objects identified by the vehicle 800 and, ifthe results do not match and the infrastructure concludes that the AI inthe vehicle 800 is malfunctioning, the server(s) 878 may transmit asignal to the vehicle 800 instructing a fail-safe computer of thevehicle 800 to assume control, notify the passengers, and complete asafe parking maneuver.

For inferencing, the server(s) 878 may include the GPU(s) 884 and one ormore programmable inference accelerators (e.g., NVIDIA's TensorRT 3).The combination of GPU-powered servers and inference acceleration maymake real-time responsiveness possible. In other examples, such as whereperformance is less critical, servers powered by CPUs, FPGAs, and otherprocessors may be used for inferencing.

Example Computing Device

FIG. 9 is a block diagram of an example computing device 900 suitablefor use in implementing some embodiments of the present disclosure.Computing device 900 may include a bus 902 that directly or indirectlycouples the following devices: memory 904, one or more centralprocessing units (CPUs) 906, one or more graphics processing units(GPUs) 908, a communication interface 910, input/output (I/O) ports 912,input/output components 914, a power supply 916, and one or morepresentation components 918 (e.g., display(s)).

Although the various blocks of FIG. 9 are shown as connected via the bus902 with lines, this is not intended to be limiting and is for clarityonly. For example, in some embodiments, a presentation component 918,such as a display device, may be considered an I/O component 914 (e.g.,if the display is a touch screen). As another example, the CPUs 906and/or GPUs 908 may include memory (e.g., the memory 904 may berepresentative of a storage device in addition to the memory of the GPUs908, the CPUs 906, and/or other components). In other words, thecomputing device of FIG. 9 is merely illustrative. Distinction is notmade between such categories as “workstation,” “server,” “laptop,”“desktop,” “tablet,” “client device,” “mobile device,” “hand-helddevice,” “game console,” “electronic control unit (ECU),” “virtualreality system,” and/or other device or system types, as all arecontemplated within the scope of the computing device of FIG. 9 .

The bus 902 may represent one or more busses, such as an address bus, adata bus, a control bus, or a combination thereof. The bus 902 mayinclude one or more bus types, such as an industry standard architecture(ISA) bus, an extended industry standard architecture (EISA) bus, avideo electronics standards association (VESA) bus, a peripheralcomponent interconnect (PCI) bus, a peripheral component interconnectexpress (PCIe) bus, and/or another type of bus.

The memory 904 may include any of a variety of computer-readable media.The computer-readable media may be any available media that may beaccessed by the computing device 900. The computer-readable media mayinclude both volatile and nonvolatile media, and removable andnon-removable media. By way of example, and not limitation, thecomputer-readable media may comprise computer-storage media andcommunication media.

The computer-storage media may include both volatile and nonvolatilemedia and/or removable and non-removable media implemented in any methodor technology for storage of information such as computer-readableinstructions, data structures, program modules, and/or other data types.For example, the memory 904 may store computer-readable instructions(e.g., that represent a program(s) and/or a program element(s), such asan operating system. Computer-storage media may include, but is notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical disk storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium which may be used to storethe desired information and which may be accessed by computing device900. As used herein, computer storage media does not comprise signalsper se.

The communication media may embody computer-readable instructions, datastructures, program modules, and/or other data types in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” mayrefer to a signal that has one or more of its characteristics set orchanged in such a manner as to encode information in the signal. By wayof example, and not limitation, the communication media may includewired media such as a wired network or direct-wired connection, andwireless media such as acoustic, RF, infrared and other wireless media.Combinations of any of the above should also be included within thescope of computer-readable media.

The CPU(s) 906 may be configured to execute the computer-readableinstructions to control one or more components of the computing device900 to perform one or more of the methods and/or processes describedherein. The CPU(s) 906 may each include one or more cores (e.g., one,two, four, eight, twenty-eight, seventy-two, etc.) that are capable ofhandling a multitude of software threads simultaneously. The CPU(s) 906may include any type of processor, and may include different types ofprocessors depending on the type of computing device 900 implemented(e.g., processors with fewer cores for mobile devices and processorswith more cores for servers). For example, depending on the type ofcomputing device 900, the processor may be an ARM processor implementedusing Reduced Instruction Set Computing (RISC) or an x86 processorimplemented using Complex Instruction Set Computing (CISC). Thecomputing device 900 may include one or more CPUs 906 in addition to oneor more microprocessors or supplementary co-processors, such as mathco-processors.

The GPU(s) 908 may be used by the computing device 900 to rendergraphics (e.g., 3D graphics). The GPU(s) 908 may include hundreds orthousands of cores that are capable of handling hundreds or thousands ofsoftware threads simultaneously. The GPU(s) 908 may generate pixel datafor output images in response to rendering commands (e.g., renderingcommands from the CPU(s) 906 received via a host interface). The GPU(s)908 may include graphics memory, such as display memory, for storingpixel data. The display memory may be included as part of the memory904. The GPU(s) 908 may include two or more GPUs operating in parallel(e.g., via a link). When combined together, each GPU 908 may generatepixel data for different portions of an output image or for differentoutput images (e.g., a first GPU for a first image and a second GPU fora second image). Each GPU may include its own memory, or may sharememory with other GPUs.

In examples where the computing device 900 does not include the GPU(s)908, the CPU(s) 906 may be used to render graphics.

The communication interface 910 may include one or more receivers,transmitters, and/or transceivers that enable the computing device 900to communicate with other computing devices via an electroniccommunication network, included wired and/or wireless communications.The communication interface 910 may include components and functionalityto enable communication over any of a number of different networks, suchas wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE,ZigBee, etc.), wired networks (e.g., communicating over Ethernet),low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and/or theInternet.

The I/O ports 912 may enable the computing device 900 to be logicallycoupled to other devices including the I/O components 914, thepresentation component(s) 918, and/or other components, some of whichmay be built in to (e.g., integrated in) the computing device 900.Illustrative I/O components 914 include a microphone, mouse, keyboard,joystick, game pad, game controller, satellite dish, scanner, printer,wireless device, etc. The I/O components 914 may provide a natural userinterface (NUI) that processes air gestures, voice, or otherphysiological inputs generated by a user. In some instances, inputs maybe transmitted to an appropriate network element for further processing.An NUI may implement any combination of speech recognition, stylusrecognition, facial recognition, biometric recognition, gesturerecognition both on screen and adjacent to the screen, air gestures,head and eye tracking, and touch recognition (as described in moredetail below) associated with a display of the computing device 900. Thecomputing device 900 may be include depth cameras, such as stereoscopiccamera systems, infrared camera systems, RGB camera systems, touchscreentechnology, and combinations of these, for gesture detection andrecognition. Additionally, the computing device 900 may includeaccelerometers or gyroscopes (e.g., as part of an inertia measurementunit (IMU)) that enable detection of motion. In some examples, theoutput of the accelerometers or gyroscopes may be used by the computingdevice 900 to render immersive augmented reality or virtual reality.

The power supply 916 may include a hard-wired power supply, a batterypower supply, or a combination thereof. The power supply 916 may providepower to the computing device 900 to enable the components of thecomputing device 900 to operate.

The presentation component(s) 918 may include a display (e.g., amonitor, a touch screen, a television screen, a heads-up-display (HUD),other display types, or a combination thereof), speakers, and/or otherpresentation components. The presentation component(s) 918 may receivedata from other components (e.g., the GPU(s) 908, the CPU(s) 906, etc.),and output the data (e.g., as an image, video, sound, etc.).

The disclosure may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program modules, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program modules including routines, programs,objects, components, data structures, etc., refer to code that performparticular tasks or implement particular abstract data types. Thedisclosure may be practiced in a variety of system configurations,including hand-held devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The disclosure mayalso be practiced in distributed computing environments where tasks areperformed by remote-processing devices that are linked through acommunications network.

As used herein, a recitation of “and/or” with respect to two or moreelements should be interpreted to mean only one element, or acombination of elements. For example, “element A, element B, and/orelement C” may include only element A, only element B, only element C,element A and element B, element A and element C, element B and elementC, or elements A, B, and C. In addition, “at least one of element A orelement B” may include at least one of element A, at least one ofelement B, or at least one of element A and at least one of element B.Further, “at least one of element A and element B” may include at leastone of element A, at least one of element B, or at least one of elementA and at least one of element B.

The subject matter of the present disclosure is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of thisdisclosure. Rather, the inventors have contemplated that the claimedsubject matter might also be embodied in other ways, to includedifferent steps or combinations of steps similar to the ones describedin this document, in conjunction with other present or futuretechnologies. Moreover, although the terms “step” and/or “block” may beused herein to connote different elements of methods employed, the termsshould not be interpreted as implying any particular order among orbetween various steps herein disclosed unless and except when the orderof individual steps is explicitly described.

What is claimed is:
 1. A method comprising: determining, using one ormore neural networks and based at least on sensor data generated usingone or more sensors of an ego-machine, one or more edges correspondingto one or more surface marking types; determining, based at least on oneor more relative locations of the one or more edges with respect to theego-machine, one or more labels associated with the one or more edges;and performing, based at least on the one or more edges and the one ormore labels, one or more operations by the ego-machine.
 2. The method ofclaim 1, wherein the determining the one or more labels associated withthe one or more edges comprises at least one of: determining that afirst edge of the one or more edges is associated with a first label ofthe one or more labels based at least on the first edge being located ata first direction relative to the ego-machine; or determining that asecond edge of the one or more edges is associated with a second labelof the one or more labels based at least on the second edge beinglocated at a second direction relative to the ego-machine.
 3. The methodof claim 1, wherein the determining the one or more edges correspondingto the one or more surface marking types comprises: determining, usingthe one or more neural networks and based at least on the sensor datagenerated using the one or more sensors of the ego-machine, one or morepoints corresponding to the one or more road marking types; anddetermining, based at least on the one or more points, the one or moreedges corresponding to the one or more road marking types.
 4. The methodof claim 3, further comprising: determining, using the one or moreneural networks and based at least on the sensor data generated usingthe one or more sensors of the ego-machine, one or more confidencescores associated with the one or more points corresponding to the oneor more surface marking types, wherein the determining the one or moreedges is further based at least on the one or more confidence scores. 5.The method of claim 1, wherein the determining the one or more edgescorresponding to the one or more surface marking types comprises:determining, using the one or more neural networks and based at least onthe sensor data generated using the one or more sensors of theego-machine, one or more lines depicted by one or more imagescorresponding to the sensor data; and determining, based at least on theone or more lines, the one or more edges corresponding to the one ormore surface marking types.
 6. The method of claim 1, furthercomprising: determining, using the one or more neural networks and basedat least on the sensor data generated using the one or more sensors ofthe ego-machine, a surface associated with the one or more surfacemarking types, wherein the performing the one or more operations by theego-machine is further based at least on the surface.
 7. The method ofclaim 1, wherein: the one or more surface marking types include at leasta first surface marking type and a second surface marking type; the oneor more edges include at least a first edge corresponding to the firstsurface marking type and a second edge corresponding to the secondsurface marking type; and the one or more labels include at least afirst label associated with the first edge and a second label associatedwith a second edge.
 8. The method of claim 1, wherein the one or moresurface marking types include one or more of a dashed line, a solidline, a yellow line, a white line, an intersection line, a crosswalkline, or a lane split line.
 9. A system comprising: one or moreprocessing units to: determine, using one or more neural networks andbased at least on sensor data generated using one or more sensors of anego-machine, one or more edges corresponding to one or more markings;determine, based at least on one or more relative locations of the oneor more edges with respect to the ego-machine, one or more labelsassociated with the one or more edges; and perform, based at least onthe one or more edges and the one or more labels, one or more operationsby the ego-machine.
 10. The system of claim 9, wherein the determinationof the one or more labels associated with the one or more edgescomprises at least one of: determining that a first edge of the one ormore edges is associated with a first label of the one or more labelsbased at least on the first edge being located at a first directionrelative to the ego-machine; or determining that a second edge of theone or more edges is associated with a second label of the one or morelabels based at least on the second edge being located at a seconddirection relative to the ego-machine.
 11. The system of claim 9,wherein the determination of the one or more edges corresponding to theone or more markings comprises: determining, using the one or moreneural networks and based at least on the sensor data generated usingthe one or more sensors of the ego-machine, one or more pointscorresponding to the one or more markings; and determining, based atleast on the one or more points, the one or more edges corresponding tothe one or more markings.
 12. The system of claim 11, wherein the one ormore processing units are further to: determine, using the one or moreneural networks and based at least on the sensor data generated usingthe one or more sensors of the ego-machine, one or more confidencescores associated with the one or more points corresponding to the oneor more markings, wherein the determination of the one or more edges isfurther based at least on the one or more confidence scores.
 13. Thesystem of claim 9, wherein the determination of the one or more edgescorresponding to the one or more markings comprises: determining, usingthe one or more neural networks and based at least on the sensor datagenerated using the one or more sensors of the ego-machine, one or morelines depicted by one or more images corresponding to the sensor data;and determining, based at least on the one or more lines, the one ormore edges corresponding to the one or more markings.
 14. The system ofclaim 9, wherein the one or more processing units are further to:determine, using the one or more neural networks and based at least onthe sensor data generated using the one or more sensors of theego-machine, a surface associated with the one or more markings, whereinthe performance of the one or more operations by the ego-machine isfurther based at least on the surface.
 15. The system of claim 9,wherein: the one or more markings include at least a first marking and asecond marking; the one or more edges include at least a first edgecorresponding to the first marking and a second edge corresponding tothe second marking; and the one or more labels include at least a firstlabel associated with the first edge and a second label associated witha second edge.
 16. The system of claim 9, wherein the one or moremarkings include one or more of a dashed line, a solid line, a yellowline, a white line, an intersection line, a crosswalk line, or a lanesplit line.
 17. The system of claim 9, wherein the system is comprisedat least one of: a control system for an autonomous or semi-autonomousmachine; a perception system for an autonomous or semi-autonomousmachine; a system for performing simulation operations; a system forperforming deep learning operations; a system implemented using an edgedevice; a system implemented using a robot; or a system implemented atleast partially using cloud computing resources.
 18. A processorcomprising: one or more processing units to cause an ego-machine toperform one or more operations based at least on one or more edgescorresponding to one or more markings and one or more labels associatedwith the one or more edges, the one or more labels being determinedbased at least on one or more relative locations of the one or moreedges with respect to the ego-machine.
 19. The processor of claim 18,wherein the one or more processing units are further to determine, usingone or more neural networks and based at least on sensor data generatedusing one or more sensors of the ego-machine, the one or more edgescorresponding to one or more markings.
 20. The processor of claim 18,wherein the processor is comprised at least one of: a control system foran autonomous or semi-autonomous machine; a perception system for anautonomous or semi-autonomous machine; a system for performingsimulation operations; a system for performing deep learning operations;a system implemented using an edge device; a system implemented using arobot; or a system implemented at least partially using cloud computingresources.